SetupInterface (execute_on)

Most user-facing objects in MOOSE inherit from the SetupInterface class. This class provides two features to objects. Foremost, it provides the the "execute_on" parameter, which, as the name suggests, dictates when the object is to be executed. Secondly, it adds virtual setup methods that allow derived classes to perform setup applications prior to execution.

Execute On

Any object inheriting from the SetupInterface, that adds the validParams<SetupInterface> within its own parameters, will have an "execute_on" parameter that can be set to various flags, the most common flags are listed below.

Execute FlagDescription
INITIALPrior to the first time step.
TIMESTEP_BEGINPrior to the solve for each time step.
NONLINEARPrior do each non-linear iteration during the solve.
LINEARPrior do each linear iteration during the solve.
TIMESTEP_ENDAfter the solve for each time step.
SUBDOMAINExecutes when the subdomain (i.e., "blocks") change during calculations.

The "execute_on" parameter can be set to a single flag or multiple flags. For example, it may be desirable to only execute an object initially because the state of the auxiliary computation does not vary. In the input file snippet below, the ElementLengthAux computation only needs to be computed initially, thus the "exeucte_on" parameter is set as such.

[AuxKernels]
  [./min]
    type = ElementLengthAux
    variable = min
    method = min
    execute_on = initial
  [../]
  [./max]
    type = ElementLengthAux
    variable = max
    method = max
    execute_on = initial
  [../]
[../]
(test/tests/auxkernels/element_length/element_length.i)

Alternatively, it is often desirable to run a computation with multiple execute flags. For example, in the input file snippet below a TimePeriod control object that is responsible for enabling in Damper object needs to be run initially and prior to each timestep to guarantee that the damper is enabled when desired.

[Controls]
  [./damping_control]
    type = TimePeriod
    disable_objects = '*::const_damp'
    start_time = 0.25
    execute_on = 'initial timestep_begin'
  [../]
[]
(test/tests/controls/time_periods/dampers/control.i)

Depending on the system these options or others will be available, since as discussed in Creating Custom Execute Flags custom flags may be added. The complete list of execution flags is provided by MOOSE are listed in the "registerExecFlags" function.










// MOOSE includes
#include "MooseApp.h"
#include "MooseRevision.h"
#include "AppFactory.h"
#include "MooseSyntax.h"
#include "MooseInit.h"
#include "Executioner.h"
#include "PetscSupport.h"
#include "Conversion.h"
#include "CommandLine.h"
#include "InfixIterator.h"
#include "MultiApp.h"
#include "MeshModifier.h"
#include "MeshGenerator.h"
#include "DependencyResolver.h"
#include "MooseUtils.h"
#include "MooseObjectAction.h"
#include "InputParameterWarehouse.h"
#include "SystemInfo.h"
#include "RestartableDataIO.h"
#include "MooseMesh.h"
#include "FileOutput.h"
#include "ConsoleUtils.h"
#include "JsonSyntaxTree.h"
#include "JsonInputFileFormatter.h"
#include "SONDefinitionFormatter.h"
#include "RelationshipManager.h"
#include "Registry.h"
#include "SerializerGuard.h"
#include "PerfGraphInterface.h" // For TIME_SECTIOn
#include "Attributes.h"

// Regular expression includes
#include "pcrecpp.h"

#include "libmesh/exodusII_io.h"
#include "libmesh/mesh_refinement.h"
#include "libmesh/string_to_enum.h"
#include "libmesh/checkpoint_io.h"

// System include for dynamic library methods
#include <dlfcn.h>
#include <sys/utsname.h> // utsname

// C++ includes
#include <numeric> // std::accumulate
#include <fstream>
#include <sys/types.h>
#include <unistd.h>
#include <cstdlib> // for system()
#include <chrono>
#include <thread>

#define QUOTE(macro) stringifyName(macro)

template <>
InputParameters
validParams<MooseApp>()
{
  InputParameters params = emptyInputParameters();

  params.addCommandLineParam<bool>(
      "display_version", "-v --version", false, "Print application version");
  params.addCommandLineParam<std::string>("input_file", "-i <input_file>", "Specify an input file");
  params.addCommandLineParam<std::string>(
      "mesh_only",
      "--mesh-only [mesh_file_name]",
      "Setup and Output the input mesh only (Default: \"<input_file_name>_in.e\")");

  params.addCommandLineParam<bool>("show_input",
                                   "--show-input",
                                   false,
                                   "Shows the parsed input file before running the simulation.");
  params.addCommandLineParam<bool>(
      "show_outputs", "--show-outputs", false, "Shows the output execution time information.");
  params.addCommandLineParam<bool>(
      "show_controls", "--show-controls", false, "Shows the Control logic available and executed.");

  params.addCommandLineParam<bool>(
      "no_color", "--no-color", false, "Disable coloring of all Console outputs.");
  params.addCommandLineParam<std::string>("color",
                                          "--color [auto,on,off]",
                                          "default-on",
                                          "Whether to use color in console output (default 'on').");

  params.addCommandLineParam<bool>("help", "-h --help", false, "Displays CLI usage statement.");
  params.addCommandLineParam<bool>(
      "minimal",
      "--minimal",
      false,
      "Ignore input file and build a minimal application with Transient executioner.");

  params.addCommandLineParam<std::string>(
      "definition", "--definition", "Shows a SON style input definition dump for input validation");
  params.addCommandLineParam<std::string>(
      "dump", "--dump [search_string]", "Shows a dump of available input file syntax.");
  params.addCommandLineParam<bool>(
      "registry", "--registry", "Lists all known objects and actions.");
  params.addCommandLineParam<bool>(
      "registry_hit", "--registry-hit", "Lists all known objects and actions in hit format.");

  params.addCommandLineParam<bool>(
      "apptype", "--type", false, "Return the name of the application object.");
  params.addCommandLineParam<std::string>(
      "yaml", "--yaml", "Dumps input file syntax in YAML format.");
  params.addCommandLineParam<std::string>(
      "json", "--json", "Dumps input file syntax in JSON format.");
  params.addCommandLineParam<bool>(
      "syntax", "--syntax", false, "Dumps the associated Action syntax paths ONLY");
  params.addCommandLineParam<bool>("check_input",
                                   "--check-input",
                                   false,
                                   "Check the input file (i.e. requires -i <filename>) and quit.");
  params.addCommandLineParam<bool>(
      "list_constructed_objects",
      "--list-constructed-objects",
      false,
      "List all moose object type names constructed by the master app factory.");

  params.addCommandLineParam<unsigned int>(
      "n_threads", "--n-threads=<n>", 1, "Runs the specified number of threads per process");

  params.addCommandLineParam<bool>(
      "warn_unused", "-w --warn-unused", false, "Warn about unused input file options");
  params.addCommandLineParam<bool>("error_unused",
                                   "-e --error-unused",
                                   false,
                                   "Error when encountering unused input file options");
  params.addCommandLineParam<bool>(
      "error_override",
      "-o --error-override",
      false,
      "Error when encountering overridden or parameters supplied multiple times");
  params.addCommandLineParam<bool>(
      "error_deprecated", "--error-deprecated", false, "Turn deprecated code messages into Errors");

  params.addCommandLineParam<bool>(
      "distributed_mesh",
      "--distributed-mesh",
      false,
      "The libMesh Mesh underlying MooseMesh should always be a DistributedMesh");

  params.addCommandLineParam<std::string>(
      "split_mesh",
      "--split-mesh [splits]",
      "comma-separated list of numbers of chunks to split the mesh into");

  params.addCommandLineParam<std::string>("split_file",
                                          "--split-file [filename]",
                                          "",
                                          "optional name of split mesh file(s) to write/read");

  params.addCommandLineParam<bool>(
      "use_split", "--use-split", false, "use split distributed mesh files");

  params.addCommandLineParam<unsigned int>(
      "refinements",
      "-r <n>",
      0,
      "Specify additional initial uniform refinements for automatic scaling");

  params.addCommandLineParam<std::string>("recover",
                                          "--recover [file_base]",
                                          "Continue the calculation.  If file_base is omitted then "
                                          "the most recent recovery file will be utilized");

  params.addCommandLineParam<std::string>("recoversuffix",
                                          "--recoversuffix [suffix]",
                                          "Use a different file extension, other than cpr, "
                                          "for a recovery file");

  params.addCommandLineParam<bool>("half_transient",
                                   "--half-transient",
                                   false,
                                   "When true the simulation will only run half of "
                                   "its specified transient (ie half the "
                                   "timesteps).  This is useful for testing "
                                   "recovery and restart");

  // No default on these two options, they must not both be valid
  params.addCommandLineParam<bool>(
      "trap_fpe",
      "--trap-fpe",
      "Enable Floating Point Exception handling in critical sections of "
      "code.  This is enabled automatically in DEBUG mode");
  params.addCommandLineParam<bool>("no_trap_fpe",
                                   "--no-trap-fpe",
                                   "Disable Floating Point Exception handling in critical "
                                   "sections of code when using DEBUG mode.");

  params.addCommandLineParam<bool>("error", "--error", false, "Turn all warnings into errors");

  params.addCommandLineParam<bool>(
      "timing",
      "-t --timing",
      false,
      "Enable all performance logging for timing purposes. This will disable all "
      "screen output of performance logs for all Console objects.");
  params.addCommandLineParam<bool>("no_timing",
                                   "--no-timing",
                                   false,
                                   "Disabled performance logging. Overrides -t or --timing "
                                   "if passed in conjunction with this flag");

  params.addCommandLineParam<bool>(
      "allow_test_objects", "--allow-test-objects", false, "Register test objects and syntax.");

  // Options ignored by MOOSE but picked up by libMesh, these are here so that they are displayed in
  // the application help
  params.addCommandLineParam<bool>(
      "keep_cout",
      "--keep-cout",
      false,
      "Keep standard output from all processors when running in parallel");
  params.addCommandLineParam<bool>(
      "redirect_stdout",
      "--redirect-stdout",
      false,
      "Keep standard output from all processors when running in parallel");

  // Options for debugging
  params.addCommandLineParam<std::string>("start_in_debugger",
                                          "--start-in-debugger <debugger>",
                                          "Start the application and attach a debugger.  This will "
                                          "launch xterm windows using the command you specify for "
                                          "'debugger'");

  params.addCommandLineParam<unsigned int>("stop_for_debugger",
                                           "--stop-for-debugger [seconds]",
                                           30,
                                           "Pauses the application during startup for the "
                                           "specified time to allow for connection of debuggers.");

  params.addPrivateParam<std::string>("_app_name"); // the name passed to AppFactory::create
  params.addPrivateParam<std::string>("_type");
  params.addPrivateParam<int>("_argc");
  params.addPrivateParam<char **>("_argv");
  params.addPrivateParam<std::shared_ptr<CommandLine>>("_command_line");
  params.addPrivateParam<std::shared_ptr<Parallel::Communicator>>("_comm");
  params.addPrivateParam<unsigned int>("_multiapp_level");
  params.addPrivateParam<unsigned int>("_multiapp_number");

  return params;
}

MooseApp::MooseApp(InputParameters parameters)
  : ConsoleStreamInterface(*this),
    ParallelObject(*parameters.get<std::shared_ptr<Parallel::Communicator>>(
        "_comm")), // Can't call getParam() before pars is set
    _name(parameters.get<std::string>("_app_name")),
    _pars(parameters),
    _type(getParam<std::string>("_type")),
    _comm(getParam<std::shared_ptr<Parallel::Communicator>>("_comm")),
    _output_position_set(false),
    _start_time_set(false),
    _start_time(0.0),
    _global_time_offset(0.0),
    _output_warehouse(*this),
    _input_parameter_warehouse(new InputParameterWarehouse()),
    _action_factory(*this),
    _action_warehouse(*this, _syntax, _action_factory),
    _parser(*this, _action_warehouse),
    _use_nonlinear(true),
    _use_eigen_value(false),
    _enable_unused_check(WARN_UNUSED),
    _factory(*this),
    _error_overridden(false),
    _ready_to_exit(false),
    _initial_from_file(false),
    _distributed_mesh_on_command_line(false),
    _recover(false),
    _restart(false),
    _split_mesh(false),
    _use_split(parameters.get<bool>("use_split")),
#ifdef DEBUG
    _trap_fpe(true),
#else
    _trap_fpe(false),
#endif
    _recover_suffix("cpr"),
    _half_transient(false),
    _check_input(getParam<bool>("check_input")),
    _restartable_data(libMesh::n_threads()),
    _multiapp_level(
        isParamValid("_multiapp_level") ? parameters.get<unsigned int>("_multiapp_level") : 0),
    _multiapp_number(
        isParamValid("_multiapp_number") ? parameters.get<unsigned int>("_multiapp_number") : 0),
    _setup_timer(_perf_graph.registerSection("MooseApp::setup", 2)),
    _setup_options_timer(_perf_graph.registerSection("MooseApp::setupOptions", 5)),
    _run_input_file_timer(_perf_graph.registerSection("MooseApp::runInputFile", 3)),
    _execute_timer(_perf_graph.registerSection("MooseApp::execute", 2)),
    _execute_executioner_timer(_perf_graph.registerSection("MooseApp::executeExecutioner", 3)),
    _restore_timer(_perf_graph.registerSection("MooseApp::restore", 2)),
    _run_timer(_perf_graph.registerSection("MooseApp::run", 3)),
    _execute_mesh_modifiers_timer(_perf_graph.registerSection("MooseApp::executeMeshModifiers", 1)),
    _execute_mesh_generators_timer(
        _perf_graph.registerSection("MooseApp::executeMeshGenerators", 1)),
    _restore_cached_backup_timer(_perf_graph.registerSection("MooseApp::restoreCachedBackup", 2)),
    _create_minimal_app_timer(_perf_graph.registerSection("MooseApp::createMinimalApp", 3))
{
  Registry::addKnownLabel(_type);
  Moose::registerAll(_factory, _action_factory, _syntax);

  _the_warehouse = libmesh_make_unique<TheWarehouse>();
  _the_warehouse->registerAttribute<AttribMatrixTags>("matrix_tags", 0);
  _the_warehouse->registerAttribute<AttribVectorTags>("vector_tags", 0);
  _the_warehouse->registerAttribute<AttribExecOns>("exec_ons", 0);
  _the_warehouse->registerAttribute<AttribSubdomains>("subdomains", 0);
  _the_warehouse->registerAttribute<AttribBoundaries>("boundaries", 0);
  _the_warehouse->registerAttribute<AttribThread>("thread", 0);
  _the_warehouse->registerAttribute<AttribPreIC>("pre_ic", 0);
  _the_warehouse->registerAttribute<AttribPreAux>("pre_aux", 0);
  _the_warehouse->registerAttribute<AttribName>("name", "dummy");
  _the_warehouse->registerAttribute<AttribSystem>("system", "dummy");
  _the_warehouse->registerAttribute<AttribVar>("variable", 0);
  _the_warehouse->registerAttribute<AttribInterfaces>("interfaces", 0);

  if (isParamValid("_argc") && isParamValid("_argv"))
  {
    int argc = getParam<int>("_argc");
    char ** argv = getParam<char **>("_argv");

    _sys_info = libmesh_make_unique<SystemInfo>(argc, argv);
  }
  if (isParamValid("_command_line"))
    _command_line = getParam<std::shared_ptr<CommandLine>>("_command_line");
  else
    mooseError("Valid CommandLine object required");

  if (_check_input && isParamValid("recover"))
    mooseError("Cannot run --check-input with --recover. Recover files might not exist");

  if (isParamValid("start_in_debugger"))
  {
    auto command = getParam<std::string>("start_in_debugger");

    Moose::out << "Starting in debugger using: " << command << std::endl;

    auto hostname = MooseUtils::hostname();

    std::stringstream command_stream;

    // This will start XTerm and print out some info first... then run the debugger
    command_stream << "xterm -e \"echo 'Rank: " << processor_id() << "  Hostname: " << hostname
                   << "  PID: " << getpid() << "'; echo ''; ";

    // Figure out how to run the debugger
    if (command.find("lldb") != std::string::npos || command.find("gdb") != std::string::npos)
      command_stream << command << " -p " << getpid();
    else
      mooseError("Unknown debugger: ",
                 command,
                 "\nIf this is truly what you meant then contact moose-users to have a discussion "
                 "about adding your debugger.");

    // Finish up the command
    command_stream << "\""
                   << " & ";

    std::string command_string = command_stream.str();
    Moose::out << "Running: " << command_string << std::endl;

    std::system(command_string.c_str());

    // Sleep to allow time for the debugger to attach
    std::this_thread::sleep_for(std::chrono::seconds(10));
  }

  if (!parameters.isParamSetByAddParam("stop_for_debugger"))
  {
    Moose::out << "\nStopping for " << getParam<unsigned int>("stop_for_debugger")
               << " seconds to allow attachment from a debugger.\n";

    Moose::out << "\nAll of the processes you can connect to:\n";
    Moose::out << "rank - hostname - pid\n";

    auto hostname = MooseUtils::hostname();

    {
      // The 'false' turns off the serialization warning
      SerializerGuard sg(_communicator, false); // Guarantees that the processors print in order
      Moose::err << processor_id() << " - " << hostname << " - " << getpid() << "\n";
    }

    Moose::out << "\nWaiting...\n" << std::endl;

    // Sleep to allow time for the debugger to attach
    std::this_thread::sleep_for(std::chrono::seconds(getParam<unsigned int>("stop_for_debugger")));
  }
}
(framework/src/base/MooseApp.C)

Modifying Execute On

When creating objects that inherit from SetupInterface it is possible to set, add, or remove available execute flags by retrieving and then modifying the ExecFlagEnum parameter. For example, consider the snippet below (see Output.C).

  ExecFlagEnum & exec_enum = params.set<ExecFlagEnum>("execute_on", true);
  exec_enum = Output::getDefaultExecFlagEnum();
  exec_enum = {EXEC_INITIAL, EXEC_TIMESTEP_END};
  params.setDocString("execute_on", exec_enum.getDocString());
(framework/src/outputs/Output.C)

First, the "execute_on" is retrieved for modification by using the "set" method. Notice, that a second boolean argument is passed to "set", this second flag enables "quite mode". Quite mode will modify the parameter silently as if the default was the modified parameter. In this case, the parameter will be listed as un-modified by the user. That is, InputParameters::isParamSetByUser returns false, if quite mode is not enabled this method would return true.

Second, the two new execution flags are added (EXEC_FINAL and EXEC_FAILED), therefore these additional options are available to all classes (all Output objects in this case) that inherit from this object.

Third, the default active flags are set to EXEC_INITIAL and EXEC_TIMESTEP_END, which are the defaults for all Output objects.

Finally, the documentation string for the "execute_on" parameter for the Output objects is update to reflect the changes made to the parameter. The ExecFlagEnum has a convenience function that generates a documentation string that includes the available options in the string.

Virtual Setup Methods

The SetupInterface includes virtual methods that correspond to the primary execute flags with MOOSE, these methods are listed in the header as shown here.

/**
* Gets called at the beginning of the simulation before this object is asked to do its job
*/
virtual void initialSetup();
/**
* Gets called at the beginning of the timestep before this object is asked to do its job
*/
virtual void timestepSetup();
/**
* Gets called just before the Jacobian is computed and before this object is asked to do its job
*/
virtual void jacobianSetup();
/**
* Gets called just before the residual is computed and before this object is asked to do its job
*/
virtual void residualSetup();
/**
* Gets called when the subdomain changes (i.e. in a Jacobian or residual loop) and before this
* object is asked to do its job
*/
virtual void subdomainSetup();
(framework/include/interfaces/SetupInterface.h)

In general, these methods should be utilized to perform "setup" procedures prior to the calls to execute for the corresponding execute flag.

note

A few of the methods were created prior to the execute flags, thus the names do not correspond but they remain as is to keep the API consistent: the "jacobianSetup" methods is called prior to the "NONLINEAR" execute flag and the "residualSetup" is called prior to the "LINEAR" execute flag.

Creating Custom Execute Flags

It is possible to create custom execute flags for an application. To create at utilize a custom execute flag the following steps should be followed.

1. Declare and Define an Execute Flag

Within your application a new global const should be declared in a header file. For example, within the LevelSetApp within MOOSE modules, there is a header (LevelSetTypes.h) that declares a new flag (EXEC_ADAPT_MESH).










#ifndef LEVELSETTYPES_H
#define LEVELSETTYPES_H

#include "Moose.h"

namespace LevelSet
{
extern const ExecFlagType EXEC_ADAPT_MESH;
extern const ExecFlagType EXEC_COMPUTE_MARKERS;
}

#endif
(modules/level_set/include/base/LevelSetTypes.h)

This new global must be defined, which occurs in the corresponding source file. When defining the new flags with a name and optionally an integer value.










// Level set includes
#include "LevelSetTypes.h"

// MOOSE includes
#include "MooseEnumItem.h"

const ExecFlagType LevelSet::EXEC_ADAPT_MESH("ADAPT_MESH");
const ExecFlagType LevelSet::EXEC_COMPUTE_MARKERS("COMPUTE_MARKERS", 1234);
(modules/level_set/src/base/LevelSetTypes.C)

2. Register the Execute Flag

After the new flag(s) are declared and defined, it must be registered with MOOSE. This is accomplished in similar fashion as object registration, simply add the newly created flag by calling registerExecFlag with the registerExecFlags function of your application.

LevelSetApp::registerExecFlags(Factory & factory)
{
  mooseDeprecated("use registerAll instead of registerExecFlags");
  registerExecFlagsInner(factory);
}
(modules/level_set/src/base/LevelSetApp.C)
note

If your application does not have a registerExecFlags function, it must be created. This can be done automatically by running the add_exec_flag_registration.py that is located in the scripts directory within MOOSE.

3. Add the Execute Flag to InputParameters

After a flag is registered, it must be made available to the object(s) in which are desired to be executed with the custom flag. This is done by adding this new flag to an existing objects valid parameters. For example, the following adds the EXEC_ADAPT_MESH flag to a Transfer object.

ExecFlagEnum & exec = params.set<ExecFlagEnum>("execute_on");
exec.addAvailableFlags(LevelSet::EXEC_ADAPT_MESH, LevelSet::EXEC_COMPUTE_MARKERS);
exec = {LevelSet::EXEC_COMPUTE_MARKERS, LevelSet::EXEC_ADAPT_MESH};
(modules/level_set/src/transfers/LevelSetMeshRefinementTransfer.C)

4. Use the Execute Flag

Depending on what type of custom computation is desired, various MOOSE execution calls accept execution flags, which will spawn calculations. For example, the LevelSetProblem contains a custom method that uses the EXEC_ADAPT_MESH flag to preform an additional MultiAppTransfer execution.

execMultiAppTransfers(LevelSet::EXEC_ADAPT_MESH, MultiAppTransfer::TO_MULTIAPP);
(modules/level_set/src/base/LevelSetProblem.C)