[rfhaque]

Description

This PR:

1. Adds a variant to experiment.py, application.py and package.py for enabling gpu-aware-mpi in remhos
2. Enables umpire in hypre and mfem
3. Enables GPU aware MPI in mfem

Adding/modifying a benchmark (docs: Adding a Benchmark)

• If package.py upstreamed to Spack is insufficient, add/modify repo/benchmark_name/package.py plus: create, self-assign, and link here a follow up issue with a link to the PR in the Spack repo.
• If application.py upstreamed to Ramble is insufficient, add/modify repo/benchmark_name/application.py plus: create, self-assign, and link here a follow up issue with a link to the PR in the Ramble repo.
• Add/modify an experiments/benchmark_name/experiment.py to define an experiment

This PR depends on

• The laghos source PR shared umpire device pool and mfem GPU-aware MPI CEED/Remhos#55 being merged
• Implementation of variable pool sizes in Remhos (similar to anticipated changes to shared device umpire pool CEED/Laghos#198)
• Remhos GPU branch being merged GPU + FOM CEED/Remhos#49
• Updating Remhos to point at the develop branch
• Cleaning up gpu-aware
• Fix CUDA build
• PR github.com/Add fom variables to adiak CEED/Remhos#57 must be merged

[pearce8]

Need to think about GPU-aware MPI

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 64.04%. Comparing base (55fd82e) to head (7407400).

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #1141      +/-   ##
===========================================
+ Coverage    62.42%   64.04%   +1.61%     
===========================================
  Files           49       49              
  Lines         3774     3774              
  Branches       326      326              
===========================================
+ Hits          2356     2417      +61     
+ Misses        1404     1343      -61     
  Partials        14       14              

see 2 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.