feat: hypre improvements #3339

victorapm · 2024-09-09T13:52:24Z

Replace internal GEOS SDC implementation with hypre's
Remove tracking of certain SDC times
Add hypre log level support
Update hypre umpire pools naming
Add missing coarse solver options in AMG
Remove unused SDC variables in GEOS data structures
Add separateComponents option to mechanics solver setup in MGR
Move addCommaSeparators to StringUtilities
Add unknowns and nonzeros info to linear solver message, e.g.:

Linear Solver | Success | Unknowns: 645,162 | Nonzeros: 42,898,950 | Iterations: 39 | Final Rel Res: 9.4042e-05 | Setup Time: 0.538 s | Solve Time: 0.933 s

Requires GEOS-DEV/thirdPartyLibs#286

…rFunctions

ryar9534 · 2024-09-10T19:03:49Z

Here is a quick test I ran on a version of SPE10 with burdens, compositional poromechanics, though with homogeneous properties, (so its just a high perm slab between two low perm boxes). Should be about 40 million DoF, and I ran on Dane.

Here were the results with develop a few weeks ago

Mpi ranks	Total Run Time (s)	Linear Solver Setup (s)	Linear Solver Solve (s)	Total GMRES Iterations (always 43 Total Newton iters)
64	2087	523	1966	1850
128	1451	343	1381	1833
256	963	234	662	1890
512	624	182	404	1974

Here are the results with the new Hypre

Mpi ranks	Total Run Time (s)	Linear Solver Setup (s)	Linear Solver Solve (s)	Total GMRES Iterations (always 43 Total Newton iters)
64	1568	233	1162	2281
128	1115	139	865	2229
256	723	85	572	2293
512	470	85	347	2366

Note that this does not double the number of nodes for each test, I will do that now

ryar9534 · 2024-09-11T16:36:26Z

Old results for heterogeneous SPE10 compositional poromechanics on CPU (dane)

Dane cores (Mpi ranks)	Total Run Time (s)	Linear Solver Setup (s)	Linear Solver Solve (s)	Time steps/Nonlin iters / Lin iters
2 (224)	2409	488	1764	24/76/3464
4 (448)	1440	380	977	24/76/3862
8 (896)	1075	447	583	24/76/4286

New results for heterogeneous SPE10 compositional poromechanics on CPU (dane)

Dane cores (Mpi ranks)	Total Run Time (s)	Linear Solver Setup (s)	Linear Solver Solve (s)	Time steps/Nonlin iters / Lin iters
2 (224)	1770	185	1425	24/76/3882
4 (448)	952	152	719	24/76/3960
8 (896)	686	225	417	24/76/4228

…/GEOS into feature/paludettomag1/hypre-sdc

…er in several functions

…/GEOS into feature/paludettomag1/hypre-sdc

paveltomin · 2024-10-04T21:03:11Z

src/coreComponents/linearAlgebra/interfaces/hypre/HypreInterface.cpp

+  HYPRE_SetUmpirePinnedPoolName( "HYPRE_PINNED" );
+#endif
+
+  HYPRE_SetLogLevel( getenv( "HYPRE_LOG_LEVEL" ) ? atoi( getenv( "HYPRE_LOG_LEVEL" ) ) : 0 );


can it be set by linear solver log level?

Yes, but I did this way because the information provided here is mainly for developers. Other libraries such as umpire and rocsparse work this way as well (using env variable). I am open to change the strategy here if needeed

sounds reasonable, thanks, no need to change

let's not use environment variables please.

I can remove this. What is the issue with env vars again?

I personally don't like them. It's hard to keep track of who sets them. Also we don't really do it for anything else in GEOS.

Here's a place where it's being used:

GEOS/src/coreComponents/mainInterface/initialization.cpp

Line 141 in cd8175d

int columns = getenv( "COLUMNS" ) ? atoi( getenv( "COLUMNS" )) : 120;

HYPRE_LOG_LEVEL is an option supposed to be used by developers only. It's very convenient to turn it on/off if it's a env var wrt a XML input. I don't see any harm with using env var specifically, but I'm happy to remove it if that's the best solution

victorapm · 2024-10-22T17:23:32Z

@paveltomin this should work for SEQ runs now

drmichaeltcvx · 2024-10-22T17:26:02Z

[like] Thomadakis, Michael reacted to your message:

…

________________________________ From: Victor A. P. Magri ***@***.***> Sent: Thursday, October 10, 2024 9:29:13 PM To: GEOS-DEV/GEOS ***@***.***> Cc: Thomadakis, Michael ***@***.***>; Mention ***@***.***> Subject: [**EXTERNAL**] Re: [GEOS-DEV/GEOS] feat: hypre improvements (PR #3339) Be aware this external email contains an attachment and/or link. Ensure the email and contents are expected. If there are concerns, please submit suspicious messages to the Cyber Intelligence Center using the Report Phishing button. We are investigating a few convergence issues with Poromechanics runs using this branch. @drmichaeltcvx<https://github.com/drmichaeltcvx>, please you can put your tests on hold until the issue is sorted out — Reply to this email directly, view it on GitHub<#3339 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AS6ZG2RPWTBYKN45GHRDVELZ23WSTAVCNFSM6AAAAABN4TL2HKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBWGA3DSOJWHA>. You are receiving this because you were mentioned.Message ID: ***@***.***>

drmichaeltcvx · 2024-10-22T17:26:45Z

[like] Thomadakis, Michael reacted to your message:

…

________________________________ From: Victor A. P. Magri ***@***.***> Sent: Tuesday, October 22, 2024 5:23:58 PM To: GEOS-DEV/GEOS ***@***.***> Cc: Thomadakis, Michael ***@***.***>; Mention ***@***.***> Subject: [**EXTERNAL**] Re: [GEOS-DEV/GEOS] feat: hypre improvements (PR #3339) Be aware this external email contains an attachment and/or link. Ensure the email and contents are expected. If there are concerns, please submit suspicious messages to the Cyber Intelligence Center using the Report Phishing button. @paveltomin<https://github.com/paveltomin> this should work for SEQ runs now — Reply to this email directly, view it on GitHub<#3339 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AS6ZG2S5RJU3QN4QOIMEZB3Z42C25AVCNFSM6AAAAABN4TL2HKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRZHA2DQMBVHE>. You are receiving this because you were mentioned.Message ID: ***@***.***>

codecov · 2024-10-22T21:16:53Z

Codecov Report

Attention: Patch coverage is 19.04762% with 68 lines in your changes missing coverage. Please review.

Project coverage is 57.51%. Comparing base (48fe317) to head (a850bd2).

Files with missing lines	Patch %	Lines
...es/hypre/mgrStrategies/MultiphasePoromechanics.hpp	0.00%	14 Missing ⚠️
...e/mgrStrategies/ThermalMultiphasePoromechanics.hpp	0.00%	14 Missing ⚠️
...onents/linearAlgebra/interfaces/hypre/HypreMGR.hpp	0.00%	8 Missing ⚠️
...ysics/CompositionalMultiphaseReservoirAndWells.cpp	44.44%	5 Missing ⚠️
...ntact/SolidMechanicsAugmentedLagrangianContact.cpp	0.00%	4 Missing ⚠️
...nts/linearAlgebra/interfaces/hypre/HypreSolver.cpp	0.00%	2 Missing ⚠️
...e/mgrStrategies/HybridSinglePhasePoromechanics.hpp	0.00%	2 Missing ⚠️
...a/interfaces/hypre/mgrStrategies/Hydrofracture.hpp	0.00%	2 Missing ⚠️
...hypre/mgrStrategies/LagrangianContactMechanics.hpp	0.00%	2 Missing ⚠️
...s/hypre/mgrStrategies/SinglePhasePoromechanics.hpp	0.00%	2 Missing ⚠️
... and 9 more

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3339      +/-   ##
===========================================
- Coverage    57.56%   57.51%   -0.06%     
===========================================
  Files         1092     1091       -1     
  Lines        97837    97816      -21     
===========================================
- Hits         56320    56259      -61     
- Misses       41517    41557      +40

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

CusiniM

I will approve it but please add an xml option for the linear solver log level instead of using that env variable.

…/GEOS into feature/paludettomag1/hypre-sdc

victorapm · 2024-10-31T16:37:37Z

@CusiniM @rrsettgast Could you do a new LC build and create the LvArray PR?

Can we put this on the merge queue?

Thanks!

victorapm · 2024-11-08T01:55:57Z

@castelletto1 @CusiniM @rrsettgast Can you guys take it from here? It seems there's nothing else I can do. Thanks!

CusiniM · 2024-11-12T23:20:43Z

@castelletto1 @CusiniM @rrsettgast Can you guys take it from here? It seems there's nothing else I can do. Thanks!

is this basically ready? I mean it clearly needs an LvArray PR and a rebaseline but is it ready otherwise?

victorapm · 2024-11-12T23:22:35Z

Right, it's ready. Needs a LvArray PR and LC build, which I can't do. Rebaseline is due to field value change (amgNumFunctions)

victorapm and others added 3 commits September 5, 2024 22:52

Update use of numeric_limits::max

a5a9e9e

Merge branch 'develop' into chore/paludettomag1/cuda-12-build

6430518

Replace logic for performing SDC in GEOS with HYPRE_BoomerAMGSetFilte…

7a67fe2

…rFunctions

victorapm self-assigned this Sep 9, 2024

victorapm and others added 13 commits September 20, 2024 15:36

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

7c38493

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

60d1df7

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

2d438b1

Add Unknowns and Nonzeros info to Linear Solver message

9974de5

Merge branch 'feature/paludettomag1/hypre-sdc' of github.com:GEOS-DEV…

55dbfa6

…/GEOS into feature/paludettomag1/hypre-sdc

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

1990070

Use GEOS_FMT

b56dd05

Add missing coarse solver options

1af9315

Remove unused code in GEOS regarding SDC approx.

1529f6f

Remove displacementFieldName since it became unused after SDC changes

0ecfdf3

Update a few MGR parameters + add separateComponents as input paramet…

08dd971

…er in several functions

Merge branch 'feature/paludettomag1/hypre-sdc' of github.com:GEOS-DEV…

65db028

…/GEOS into feature/paludettomag1/hypre-sdc

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

d64aae2

victorapm marked this pull request as ready for review October 4, 2024 20:12

victorapm requested review from CusiniM, paveltomin, castelletto1, rrsettgast and matteofrigo5 as code owners October 4, 2024 20:12

Update TPL tag

0cae3ff

paveltomin reviewed Oct 4, 2024

View reviewed changes

paveltomin approved these changes Oct 4, 2024

View reviewed changes

victorapm added 2 commits October 4, 2024 19:38

Turn on SDC approx. for fracture models

ecfe53c

Uncrustify

5c1b4d8

victorapm added the ci: run code coverage enables running of the code coverage CI jobs label Oct 22, 2024

Added missing SDC flag for poromechanics with wells

e0bb7a8

CusiniM approved these changes Oct 28, 2024

View reviewed changes

victorapm and others added 4 commits October 30, 2024 20:16

Reset coarsest AMG level parameters

54c455c

Update TPL tag

ac1fb3c

Uncrustify

cde7e90

Merge branch 'feature/paludettomag1/hypre-sdc' of github.com:GEOS-DEV…

d7f26f1

…/GEOS into feature/paludettomag1/hypre-sdc

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

cc223c2

paveltomin mentioned this pull request Nov 1, 2024

refactor: cleanup setMGRStrategy a bit #3422

Open

victorapm and others added 4 commits November 1, 2024 13:08

Use SDC for hydrofrac MGR strategy

4a353e6

Better robustness and performance for MGR on GPUs

44eb9b5

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

1237e99

Rebaseline

6f853d7

victorapm requested review from jhuang2601, frankfeifan, ryar9534, Guotong-Ren and cssherman as code owners November 7, 2024 23:23

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

a850bd2

victorapm added the flag: requires rebaseline Requires rebaseline branch in integratedTests label Nov 8, 2024

Merge branch 'develop' into feature/paludettomag1/hypre-sdc

8a64a0a

CusiniM mentioned this pull request Nov 12, 2024

Update to hypre v2.32.0-4-gc893886d1 GEOS-DEV/thirdPartyLibs#286

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: hypre improvements #3339

feat: hypre improvements #3339

victorapm commented Sep 9, 2024 •

edited

Loading

ryar9534 commented Sep 10, 2024 •

edited

Loading

ryar9534 commented Sep 11, 2024 •

edited

Loading

paveltomin Oct 4, 2024

victorapm Oct 4, 2024 •

edited

Loading

paveltomin Oct 4, 2024

CusiniM Oct 28, 2024

victorapm Oct 28, 2024

CusiniM Oct 28, 2024

victorapm Oct 28, 2024 •

edited

Loading

victorapm commented Oct 22, 2024

drmichaeltcvx commented Oct 22, 2024 via email

drmichaeltcvx commented Oct 22, 2024 via email

codecov bot commented Oct 22, 2024 •

edited

Loading

CusiniM left a comment

victorapm commented Oct 31, 2024

victorapm commented Nov 8, 2024

CusiniM commented Nov 12, 2024

victorapm commented Nov 12, 2024

feat: hypre improvements #3339

Are you sure you want to change the base?

feat: hypre improvements #3339

Conversation

victorapm commented Sep 9, 2024 • edited Loading

ryar9534 commented Sep 10, 2024 • edited Loading

ryar9534 commented Sep 11, 2024 • edited Loading

paveltomin Oct 4, 2024

Choose a reason for hiding this comment

victorapm Oct 4, 2024 • edited Loading

Choose a reason for hiding this comment

paveltomin Oct 4, 2024

Choose a reason for hiding this comment

CusiniM Oct 28, 2024

Choose a reason for hiding this comment

victorapm Oct 28, 2024

Choose a reason for hiding this comment

CusiniM Oct 28, 2024

Choose a reason for hiding this comment

victorapm Oct 28, 2024 • edited Loading

Choose a reason for hiding this comment

victorapm commented Oct 22, 2024

drmichaeltcvx commented Oct 22, 2024 via email

drmichaeltcvx commented Oct 22, 2024 via email

codecov bot commented Oct 22, 2024 • edited Loading

Codecov Report

CusiniM left a comment

Choose a reason for hiding this comment

victorapm commented Oct 31, 2024

victorapm commented Nov 8, 2024

CusiniM commented Nov 12, 2024

victorapm commented Nov 12, 2024

victorapm commented Sep 9, 2024 •

edited

Loading

ryar9534 commented Sep 10, 2024 •

edited

Loading

ryar9534 commented Sep 11, 2024 •

edited

Loading

victorapm Oct 4, 2024 •

edited

Loading

victorapm Oct 28, 2024 •

edited

Loading

codecov bot commented Oct 22, 2024 •

edited

Loading