Compute efficiencies

Geochemistry simulations use a large amount of memory since they store information about a complete geochemical system at each finite-element node, and, optionally, populate a huge number of AuxVariables with useful information. On the other hand, ignoring transport, they are almost embarrassingly parallel, so may be solved efficiently using a large number of processors.

This page explores computational aspects of the 2D GeoTES simulation, the Weber-Tensleep GeoTES simulation and the FORGE geothermal simulation which are coupled porous-flow + geochemistry simulations.

Memory usage for pure geochemistry simulations

While spatially-dependent geochemistry simulations are rarely run without transport, it is useful to quantify the amount of memory used by geochemistry models alone (without transport) because memory consumption can easily swamp all other considerations. Table 1 lists the numbers of species in the two simulations used in this section.

The Geochemistry Actions, such as the SpatialReactionSolver, add a large number of AuxVariables by default. These record the species' molality, free mass, free volume, activity, bulk composition, etc, which may be useful but add considerably to the memory consumption. Some examples are quantified in Table 2 and Table 3.

The results are summarised in the approximate formula where is the number of nodes, the number of species ( and in the two examples) and the number of AuxVariables.

1. The first term is largely independent of geochemistry so is not too important here.

2. The second term depends on the size of the geochemical system, , and quantifies the cost of keeping track of an entire geochemical at each node, which is approximately bytes per node. Recall that is the total number of species in the geochemical model.

3. The third term depends on the size of the Auxiliary system, which is approximately bytes per node. Recall that this is about if all the default AuxVariables are added by the Action.

Of course, the memory consumption will be dependent on the compiler and architecture, but the above formula should provide an approximate guide.

Compute time for pure geochemistry simulations

The geochemistry-only part of the FORGE geothermal simulation (aquifer_geochemistry.i with 15251 nodes and no exodus output) may be run using a different number of processors in order to demonstrate the computational efficiencies of pure geochemistry simulations that have no transport. (In fact, exactly the same computations are performed at each node in this case, but MOOSE does not know this.) As is evident from Table 4 and Figure 1, geochemical-only simulations scale very well with the number of processors.

Figure 1: CPU time required to solve the FORGE geochemistry-only simulation with no transport.

Relative compute-time spent in transport and reactions

It is common that the "reaction" part of reactive-transport simulations proceeds quickly compared with the transport part. This is because the geochemistry module bypasses a lot of MOOSE overhead and because there are no spatial derivatives that require the use of PETSc. Table 5 quantifies this for the FORGE example and an example from the geochemistry test suite. Although this is just one example, the large cost of a full PorousFlow solve is evident. If possible, compute time will be reduced by pre-computing the Darcy velocity and time-dependent temperature field and using these in the GeochemistryTimeDerivative and ConservativeAdvection Kernels, as in the advection_1.i input file.

Solver choices for simulations coupled with PorousFlow

Geochemistry simulations are often coupled with PorousFlow in order to simulate reactive transport. The previous section demonstrated that the "transport" part of such simulations is likely to be much more expensive than the "reactive" part. Hence, it is useful to explore ways of improving its efficiency. One way is to pre-compute the Darcy velocity and temperature field, and use the GeochemistryTimeDerivative and ConservativeAdvection Kernels. However, this is often not possible. Therefore, this section quantifies the compute time required for a typical PorousFlow simulation.

    type = GeneratedMeshGenerator
    dim = 2
    nx = 135
    ny = 120
    xmin = -400
    xmax = 500
    ymin = -400
    ymax = 400

[Preconditioning]
  [FSP]
    type = FSP
    topsplit = 'up' # 'up' should match the following block name
    [up]
      splitting = 'u p' # 'u' and 'p' are the names of subsolvers, below
      splitting_type  = multiplicative
      petsc_options_iname = '-pc_fieldsplit_schur_fact_type  -pc_fieldsplit_schur_precondition'
      petsc_options_value = 'full selfp'
    []
    [u]
      vars = 'f_H f_Na f_K f_Ca f_Mg f_SiO2 f_Al f_Cl f_SO4 f_HCO3 temperature'
      petsc_options_iname = '-pc_type -ksp_type'
      petsc_options_value = ' hypre    gmres'
    []
    [p]
      vars = 'porepressure'
      petsc_options_iname = '-pc_type -ksp_type'
      petsc_options_value = ' hypre    gmres'
    []
  []
[]

    type = GeneratedMeshGenerator
    dim = 3
    nx = 36
    ny = 32
    nz = 10
    xmin = -400
    xmax = 500
    ymin = -400
    ymax = 400

Reactive-transport using multiple processors

A large reactive-transport simulation based on the 2D FORGE model is used to study the scaling behaviour with the number of processors. Given the above results, this is mainly testing the scaling behaviour of the linear solver for the PorousFlow simulation. The mesh used is

    type = GeneratedMeshGenerator
    dim = 2
    nx = 450
    ny = 400
    xmin = -400
    xmax = 500
    ymin = -400
    ymax = 400

The magnitude of the problem is tabulated in Table 8. The simulation is run for 100 time-steps using the hypre boomeramg preconditioner in porous_flow.i.

The results are tabulated in Table 9 and Figure 1

Figure 2: CPU time required to solve the FORGE reactive-transport simulation involving a MultiApp coupling between PorousFlow and Geochemistry.