Stochastic Tools Batch Mode

The SamplerFullSolveMultiApp and SamplerTransientMultiApp are capable of running sub-applications in one of three different modes:

  1. normal: One sub-application is created for each row of data (num_rows) supplied by the Sampler object.

  2. batch-reset: sub-applications are created, where the sub-applications are destroyed and re-created (on the same existing MPI communicator) for each row of data supplied by the Sampler object.

  3. batch-restore: sub-applications are created, where the sub-application is backed up after initialization. Then for each row of data supplied by the Sampler object the sub-application is restored to the initial state prior to execution.

For the two "batch" options, indicates the number of applications created, which in the most general expression is given by

where is the num_rows parameter on the Sampler object, is the number of processors (launched to the mpiexec command), and is the min_procs_per_app, or the minimum number of processors which should be used to run each sub-application. If you launch your Stochastic Tools main application with fewer than min_procs_per_app, the simulation will still proceed, but will just use the maximum number of ranks you have provided as the "effective" min_procs_per_app. We can also illustrate this with an example. Table 1 shows the number of applications created () for two different choices of num_rows, min_procs_per_app, and number of processors.

Table 1: Number of sub-apps launched for the "batch" modes for different example choices of num_rows and min_procs_per_app

num_rowsmin_procs_per_appProcessorsNumber of Sub-Apps
4294
20294
4231
4211

All three modes are available when using SamplerFullSolveMultiApp, the "batch-reset" mode is not available for SamplerTransientMultiApp because the sub-application has state that must be maintained as simulation time progresses.

The primary benefit to using a batch mode is to improve performance of a simulation by reducing the memory of the running application. The performance gains depend on the type of sub-application being executed as well as the number of samples being evaluated. The following sections highlight the the performance improvements that may be expected for full solve and transient sub-applications.

Example 1: Full Solve Sub-Application

The first example demonstrates the performance improvements to expect when using SamplerFullSolveMultiApp with sub-applications. In this case, the sub-application solves steady-state diffusion on a unit cube domain with Dirichlet boundary conditions on the left, , and right, , sides of the domain, the complete input file for this problem is given in Listing 1.

Listing 1: Complete input file for steady-state diffusion problem.

[Mesh<<<{"href": "../../syntax/Mesh/index.html"}>>>]
  type = GeneratedMesh
  dim = 3
  nx = 10
  ny = 10
  nz = 10
[]

[Variables<<<{"href": "../../syntax/Variables/index.html"}>>>]
  [u]
  []
[]

[Kernels<<<{"href": "../../syntax/Kernels/index.html"}>>>]
  [diff]
    type = ADDiffusion<<<{"description": "Same as `Diffusion` in terms of physics/residual, but the Jacobian is computed using forward automatic differentiation", "href": "../../source/kernels/ADDiffusion.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
  []
  [time]
    type = ADTimeDerivative<<<{"description": "The time derivative operator with the weak form of $(\\psi_i, \\frac{\\partial u_h}{\\partial t})$.", "href": "../../source/kernels/ADTimeDerivative.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
  []
[]

[BCs<<<{"href": "../../syntax/BCs/index.html"}>>>]
  [left]
    type = DirichletBC<<<{"description": "Imposes the essential boundary condition $u=g$, where $g$ is a constant, controllable value.", "href": "../../source/bcs/DirichletBC.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
    boundary<<<{"description": "The list of boundary IDs from the mesh where this object applies"}>>> = left
    value<<<{"description": "Value of the BC"}>>> = 0
  []
  [right]
    type = DirichletBC<<<{"description": "Imposes the essential boundary condition $u=g$, where $g$ is a constant, controllable value.", "href": "../../source/bcs/DirichletBC.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
    boundary<<<{"description": "The list of boundary IDs from the mesh where this object applies"}>>> = right
    value<<<{"description": "Value of the BC"}>>> = 1
  []
[]

[Postprocessors<<<{"href": "../../syntax/Postprocessors/index.html"}>>>]
  [average]
    type = AverageNodalVariableValue<<<{"description": "Computes the average value of a field by sampling all nodal solutions on the domain or within a subdomain", "href": "../../source/postprocessors/AverageNodalVariableValue.html"}>>>
    variable<<<{"description": "The name of the variable that this postprocessor operates on"}>>> = u
  []
[]

[Executioner<<<{"href": "../../syntax/Executioner/index.html"}>>>]
  type = Transient
  num_steps = 1
  dt = 0.25
  solve_type = NEWTON
[]

[Controls<<<{"href": "../../syntax/Controls/index.html"}>>>]
  [receiver]
    type = SamplerReceiver<<<{"description": "Control for receiving data from a Sampler via SamplerParameterTransfer.", "href": "../../source/controls/SamplerReceiver.html"}>>>
  []
[]

[Outputs<<<{"href": "../../syntax/Outputs/index.html"}>>>]
[]
(modules/stochastic_tools/examples/batch/sub.i)

The master application does not perform a solve, it performs a stochastic analysis using the MonteCarlo object to perturb the values of the two Dirichlet conditions on the sub-applications to vary with a uniform distribution. The complete input file for the master application is given in Listing 1.

Listing 2: Complete input file for master application that performs stochastic simulations of the steady-state diffusion problem in Listing 1 using Monte Carlo sampling.

[StochasticTools<<<{"href": "../../syntax/StochasticTools/index.html"}>>>]
[]

[Distributions<<<{"href": "../../syntax/Distributions/index.html"}>>>]
  [uniform]
    type = Uniform<<<{"description": "Continuous uniform distribution.", "href": "../../source/distributions/Uniform.html"}>>>
    lower_bound<<<{"description": "Distribution lower bound"}>>> = 1
    upper_bound<<<{"description": "Distribution upper bound"}>>> = 9
  []
[]

[Samplers<<<{"href": "../../syntax/Samplers/index.html"}>>>]
  [mc]
    type = MonteCarlo<<<{"description": "Monte Carlo Sampler.", "href": "../../source/samplers/MonteCarloSampler.html"}>>>
    num_rows<<<{"description": "The number of rows per matrix to generate."}>>> = 10
    distributions<<<{"description": "The distribution names to be sampled, the number of distributions provided defines the number of columns per matrix."}>>> = 'uniform uniform'
  []
[]

[MultiApps<<<{"href": "../../syntax/MultiApps/index.html"}>>>]
  [runner]
    type = SamplerFullSolveMultiApp<<<{"description": "Creates a full-solve type sub-application for each row of each Sampler matrix.", "href": "../../source/multiapps/SamplerFullSolveMultiApp.html"}>>>
    sampler<<<{"description": "The Sampler object to utilize for creating MultiApps."}>>> = mc
    input_files<<<{"description": "The input file for each App.  If this parameter only contains one input file it will be used for all of the Apps.  When using 'positions_from_file' it is also admissable to provide one input_file per file."}>>> = 'sub.i'
    mode<<<{"description": "The operation mode, 'normal' creates one sub-application for each row in the Sampler and 'batch-reset' and 'batch-restore' creates N sub-applications, where N is the minimum of 'num_rows' in the Sampler and floor(number of processes / min_procs_per_app). To run the rows in the Sampler, 'batch-reset' will destroy and re-create sub-apps as needed, whereas the 'batch-restore' will backup and restore sub-apps to the initial state prior to execution, without destruction."}>>> = batch-restore
  []
[]

[Transfers<<<{"href": "../../syntax/Transfers/index.html"}>>>]
  [runner]
    type = SamplerParameterTransfer<<<{"description": "Copies Sampler data to a SamplerReceiver object.", "href": "../../source/transfers/SamplerParameterTransfer.html"}>>>
    to_multi_app<<<{"description": "The name of the MultiApp to transfer the data to"}>>> = runner
    parameters<<<{"description": "A list of parameters (on the sub application) to control with the Sampler data. The order of the parameters listed here should match the order of the items in the Sampler."}>>> = 'BCs/left/value BCs/right/value'
    sampler<<<{"description": "A the Sampler object that Transfer is associated.."}>>> = mc
  []
  [data]
    type = SamplerPostprocessorTransfer<<<{"description": "Transfers data from Postprocessors on the sub-application to a VectorPostprocessor on the master application.", "href": "../../source/transfers/SamplerPostprocessorTransfer.html"}>>>
    from_multi_app<<<{"description": "The name of the MultiApp to receive data from"}>>> = runner
    to_vector_postprocessor<<<{"description": "The name of the VectorPostprocessor in the MultiApp to transfer values to."}>>> = storage
    from_postprocessor<<<{"description": "The name(s) of the Postprocessor(s) on the sub-app to transfer from."}>>> = average
    sampler<<<{"description": "A the Sampler object that Transfer is associated.."}>>> = mc
  []
[]

[VectorPostprocessors<<<{"href": "../../syntax/VectorPostprocessors/index.html"}>>>]
  [storage]
    type = StochasticResults<<<{"description": "Storage container for stochastic simulation results coming from a Postprocessor.", "href": "../../source/vectorpostprocessors/StochasticResults.html"}>>>
  []
[]

[Postprocessors<<<{"href": "../../syntax/Postprocessors/index.html"}>>>]
  [total]
    type = MemoryUsage<<<{"description": "Memory usage statistics for the running simulation.", "href": "../../source/postprocessors/MemoryUsage.html"}>>>
    execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = 'INITIAL TIMESTEP_END'
  []
  [per_proc]
    type = MemoryUsage<<<{"description": "Memory usage statistics for the running simulation.", "href": "../../source/postprocessors/MemoryUsage.html"}>>>
    value_type<<<{"description": "Aggregation method to apply to the requested memory metric."}>>> = "average"
    execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = 'INITIAL TIMESTEP_END'
  []
  [max_proc]
    type = MemoryUsage<<<{"description": "Memory usage statistics for the running simulation.", "href": "../../source/postprocessors/MemoryUsage.html"}>>>
    value_type<<<{"description": "Aggregation method to apply to the requested memory metric."}>>> = "max_process"
    execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = 'INITIAL TIMESTEP_END'
  []
[]

[Outputs<<<{"href": "../../syntax/Outputs/index.html"}>>>]
  csv<<<{"description": "Output the scalar variable and postprocessors to a *.csv file using the default CSV output."}>>> = true
  perf_graph<<<{"description": "Enable printing of the performance graph to the screen (Console)"}>>> = true
[]
(modules/stochastic_tools/examples/batch/full_solve.i)

The example is executed to demonstrate memory performance of the various modes of operation: "normal", "batch-reset", and "batch-restore". Each mode is executed with increasing number of Monte Carlo samples by setting the "n_samples" parameter of the MonteCarloSampler object. Figure 1 and Figure 2 show the resulting memory use at the end of the simulation for each mode of operation with increasing sample numbers in serial and in parallel, respectively.

Figure 1: Total memory at the end of the simulation using a SamplerFullSolveMultiApp with increasing number of Monte Carlo samples for the three available modes of operation running on a single processor.

Figure 2: Total memory and maximum memory per processor at the end of the simulation using a SamplerFullSolveMultiApp with increasing number of Monte Carlo samples for the three available modes of operation running on 56 processors.

An important feature of the various modes of operation is that run-time is not negatively impacted by changing the mode, in some cases using a batch mode can actually decrease total simulation run time.

The total run time results for the full solve problem in serial and parallel are shown in Figure 3 and Figure 4, respectively. The time shown in these plots is the total simulation time, which encompasses both the simulation initialization and solve. The differences in speed are mainly due to the installation and destruction of the sub-application. When running in 'batch-reset' mode, each data sample causes the sub-application to be created and destroyed during the solve, causing the slowest performance. The 'normal' mode creates all sub-applications up front, and the 'batch-restore' method uses the backup-restore capability to save the state of the sub-applications, thus does not require as many instantiations and has the lowest run-time. For this example, the solve portion is minimal as such the sub-application creation time plays a large role. As the solve time increases time gains can be expected to be minimal.

Figure 3: Total execution time of a simulation using SamplerFullSolveMultiApp with increasing number of Monte Carlo samples for the available modes of operation on a single processor.

Figure 4: Total execution time of a simulation using SamplerFullSolveMultiApp with increasing number of Monte Carlo samples for the available modes of operation on 56 processors.

Example 2: Transient Sub-Application

The second example is nearly identical to the first, except the master application is a transient solve that sets the boundary conditions at the end of each time step. The only difference occurs in the master input file, in the Executioner and MultiApps block, as shown in Listing 3.

Listing 3: Complete input file for a transient master application that performs stochastic simulations of a diffusion problem with time varying boundary conditions using Monte Carol sampling.

[Executioner<<<{"href": "../../syntax/Executioner/index.html"}>>>]
  type = Transient
  num_steps = 10
[]

[MultiApps<<<{"href": "../../syntax/MultiApps/index.html"}>>>]
  [runner]
    type = SamplerFullSolveMultiApp<<<{"description": "Creates a full-solve type sub-application for each row of each Sampler matrix.", "href": "../../source/multiapps/SamplerFullSolveMultiApp.html"}>>>
    sampler<<<{"description": "The Sampler object to utilize for creating MultiApps."}>>> = mc
    input_files<<<{"description": "The input file for each App.  If this parameter only contains one input file it will be used for all of the Apps.  When using 'positions_from_file' it is also admissable to provide one input_file per file."}>>> = 'sub.i'
    execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = 'INITIAL TIMESTEP_END'
    mode<<<{"description": "The operation mode, 'normal' creates one sub-application for each row in the Sampler and 'batch-reset' and 'batch-restore' creates N sub-applications, where N is the minimum of 'num_rows' in the Sampler and floor(number of processes / min_procs_per_app). To run the rows in the Sampler, 'batch-reset' will destroy and re-create sub-apps as needed, whereas the 'batch-restore' will backup and restore sub-apps to the initial state prior to execution, without destruction."}>>> = batch-restore
  []
[]
(modules/stochastic_tools/examples/batch/transient.i)

The results shown in Figure 5 and Figure 6 include the memory use at the end of the simulation (10 time steps) for each mode of operation within increasing number of samples in serial and parallel. Recall, as mentioned above, that the "batch-reset" mode is not available in the SamplerTransientMultiApp.

Figure 5: Total memory at the end of the simulation using a SamplerTransientMultiApp with increasing number of Monte Carlo samples for the two available modes of operation running on a single processor.

Figure 6: Total memory and maximum memory per processor at the end of the simulation using a SamplerTransientMultiApp with increasing number of Monte Carlo samples for the two available modes of operation running on 56 processors.

Again, an important feature of the various modes of operation is that run-time is not negatively impacted by changing the mode as seen in Figure 7 and Figure 8. The solve portion of this example is significantly longer than the steady-state example. As such the differences in execution time due to the instantiating of objects is diminished and both modes behave similarly.

Figure 7: Total execution time of a simulation using SamplerTransientMultiApp with increasing number of Monte Carlo samples for the available modes of operation on a single processor.

Figure 8: Total execution time of a simulation using SamplerTransientMultiApp with increasing number of Monte Carlo samples for the available modes of operation on 56 processors.