LibtorchDRLControl

Sets the value of multiple 'Real' input parameters and postprocessors based on a Deep Reinforcement Learning (DRL) neural network trained using a PPO algorithm.

Overview

This object is the runtime policy executor paired with LibtorchDRLControlTrainer. It extends LibtorchNeuralNetControl with stochastic policy sampling, action reuse, optional smoothing, and restartable policy state. For deterministic execution of the same actor, set "stochastic" to false. Use LibtorchNeuralNetControl instead when a plain deterministic neural-net control object is preferred without the DRL-specific execution features.

Execution Model

LibtorchDRLControl only supports TIMESTEP_BEGIN. On each execution it reads the current observation values, combines them with the stored history implied by "input_timesteps", and evaluates the actor when a new policy action is needed.

If "stochastic" is true, the action is sampled from the actor distribution and the corresponding log probabilities are stored for PPO training. If it is false, the deterministic actor output is used instead. The policy evaluation can be reused across multiple time steps with "num_steps_in_period". For num_steps_in_period = P, the actor is evaluated on steps $var element = document.getElementById("moose-equation-545e5e3e-591e-484d-9570-155279ab1c28");katex.render("1, P+1, 2P+1, \\ldots", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ , and the action is held in between. A transfer that executes the control before the first time step also triggers an evaluation at step 0. The observation history only advances when the actor is evaluated, so consecutive entries of the stacked input are one control period apart. Set "timestep_window" on the trainer to the same value so that it rebuilds the observations the controller used.

The applied control can also be smoothed exponentially with "control_smoothing_factor": $var element = document.getElementById("moose-equation-a295b447-8970-44c3-80ae-4bc1ad60dc45");katex.render("u_t^{\\mathrm{applied}} = u_{t-1}^{\\mathrm{applied}} + \\alpha\\left(u_t^{\\mathrm{policy}} - u_{t-1}^{\\mathrm{applied}}\\right),", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ where $var element = document.getElementById("moose-equation-ea48869e-8420-47e9-a887-615c3b2f26f1");katex.render("\\alpha \\in (0, 1]", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ is the control_smoothing_factor value. Setting control_smoothing_factor = 1 applies the raw policy action directly. Because each applied value lies between the previous applied value and the policy action, smoothing does not overshoot the action or leave the range set by "min_control_value" and "max_control_value", provided the starting value lies in that range. The smoothed signal starts from zero, so with $var element = document.getElementById("moose-equation-54139ed5-6322-44ba-b743-042ac2f822bf");katex.render("\\alpha < 1", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ the first applied values lie between zero and the policy action. The smoothing is applied every time step, including the steps within a control period that reuse the previous action. LibtorchControlValuePostprocessor reports the policy action $var element = document.getElementById("moose-equation-c68868c8-6280-40f0-b9cf-b25349e102bc");katex.render("u_t^{\\mathrm{policy}}", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ , not the applied value.

The controller stores the observation history, smoothed signal, and the libtorch CPU random-number-generator state as restartable data. This keeps stochastic recovered runs aligned with uninterrupted runs, provided the same controller state is recovered.

Input Parameters

observationsPostprocessors used as the current observation vector.
C++ Type:std::vector<PostprocessorName>
Unit:(no unit assumed)
Controllable:No
Description:Postprocessors used as the current observation vector.
parametersControllable input parameters driven by the network.
C++ Type:std::vector<std::string>
Controllable:No
Description:Controllable input parameters driven by the network.

Required Parameters

action_scaling_factorsPer-action scaling embedded in the controller outputs so saved checkpoints stay in physical units.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Per-action scaling embedded in the controller outputs so saved checkpoints stay in physical units.
activation_functionrelu Activation name for each hidden layer, or one shared value for all layers.
Default:relu
C++ Type:std::vector<std::string>
Controllable:No
Description:Activation name for each hidden layer, or one shared value for all layers.
control_offsetsOptional constant offsets added to each policy signal before setting the controlled parameter. This can be used to train a policy on deviations from a nominal signal.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional constant offsets added to each policy signal before setting the controlled parameter. This can be used to train a policy on deviations from a nominal signal.
control_smoothing_factor1Exponential smoothing factor for the applied control signal: the fraction of the change from the previously applied signal toward the new policy action that is applied at each time step. A value of 1 applies the policy action directly.
Default:1
C++ Type:Real
Unit:(no unit assumed)
Range:control_smoothing_factor>0 & control_smoothing_factor<=1
Controllable:No
Description:Exponential smoothing factor for the applied control signal: the fraction of the change from the previously applied signal toward the new policy action that is applied at each time step. A value of 1 applies the policy action directly.
depends_onThe Controls that this control relies upon (i.e. must execute before this one)
C++ Type:std::vector<std::string>
Controllable:No
Description:The Controls that this control relies upon (i.e. must execute before this one)
execute_onTIMESTEP_BEGINThe list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
Default:TIMESTEP_BEGIN
C++ Type:ExecFlagEnum
Options:XFEM_MARK, FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, LINEAR_CONVERGENCE, NONLINEAR, NONLINEAR_CONVERGENCE, POSTCHECK, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, MULTIAPP_FIXED_POINT_CONVERGENCE, MULTISYSTEM_FIXED_POINT_ITERATION_END, FINAL, CUSTOM, PRE_MULTIAPP_SETUP
Controllable:No
Description:The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
filenameFile containing the controller network.
C++ Type:std::string
Controllable:No
Description:File containing the controller network.
input_timesteps1Number of recent timesteps to stack into each network input.
Default:1
C++ Type:unsigned int
Range:1<=input_timesteps
Controllable:No
Description:Number of recent timesteps to stack into each network input.
max_control_valueOptional upper bounds for each control signal.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional upper bounds for each control signal.
min_control_valueOptional lower bounds for each control signal.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional lower bounds for each control signal.
num_neurons_per_layerHidden-layer widths used when constructing the controller.
C++ Type:std::vector<unsigned int>
Controllable:No
Description:Hidden-layer widths used when constructing the controller.
num_steps_in_period1Number of time steps between policy evaluations. The observation history advances once per period, so the trainer's 'timestep_window' should match this value.
Default:1
C++ Type:unsigned int
Range:1<=num_steps_in_period
Controllable:No
Description:Number of time steps between policy evaluations. The observation history advances once per period, so the trainer's 'timestep_window' should match this value.
observation_scaling_factorsOptional multipliers applied after shifting the observation values.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional multipliers applied after shifting the observation values.
observation_shift_factorsOptional offsets applied to the observation values before scaling.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional offsets applied to the observation values before scaling.
seedSeed for the random number generator.
C++ Type:unsigned int
Controllable:No
Description:Seed for the random number generator.
state_independent_stdTrueIf true, interpret the unbounded Gaussian actor as learning one log-std per action dimension. If false, use a state-dependent std head.
Default:True
C++ Type:bool
Controllable:No
Description:If true, interpret the unbounded Gaussian actor as learning one log-std per action dimension. If false, use a state-dependent std head.
stochasticTrueIf true, sample from the policy distribution; otherwise use the deterministic action.
Default:True
C++ Type:bool
Controllable:No
Description:If true, sample from the policy distribution; otherwise use the deterministic action.

Optional Parameters

control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector<std::string>
Controllable:No
Description:Adds user-defined labels for accessing object parameters via control logic.
enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Controllable:No
Description:Set the enabled status of the MooseObject.
implicitTrueDetermines whether this object is calculated using an implicit or explicit form
Default:True
C++ Type:bool
Controllable:No
Description:Determines whether this object is calculated using an implicit or explicit form

Advanced Parameters

Input Files

(modules/stochastic_tools/test/tests/controls/libtorch_drl_control/libtorch_drl_control.i)
(modules/stochastic_tools/test/tests/transfers/sampler_drl_control_transfer/sub.i)
(modules/stochastic_tools/test/tests/transfers/libtorch_nn_transfer/libtorch_drl_control_sub.i)
(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_sub.i)

(modules/stochastic_tools/test/tests/controls/libtorch_drl_control/libtorch_drl_control.i)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 1
    xmin = 0.0
    xmax = 7.0
    nx = 20
  []
[]

[Variables]
  [temp]
    initial_condition = 300
  []
[]

[Kernels]
  [time]
    type = CoefTimeDerivative
    variable = temp
    Coefficient = '${fparse 1.00630182*1.225}'
  []
  [heat_conduc]
    type = MatDiffusion
    variable = temp
    diffusivity = 'k'
  []
[]

[BCs]
  [left_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'left'
    variable = temp
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = temp
    boundary = 'right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    expression = '15.0*sin(t/86400.0 *pi) + 273.0'
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = 26.53832364
  []
[]

[Postprocessors]
  [center_temp]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [center_temp_tend]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [env_temp]
    type = FunctionValuePostprocessor
    function = temp_env
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [left_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [log_prob_left_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
[]

[Controls]
  inactive = src_control_empty
  [src_control]
    seed = 11
    type = LibtorchDRLControl
    parameters = "BCs/left_flux/value"
    observations = 'center_temp env_temp'

    input_timesteps = 2
    observation_scaling_factors = '0.03 0.03'
    observation_shift_factors = '270 270'
    action_scaling_factors = 100

    filename = 'mynet_control.net'
    num_neurons_per_layer = '4 2'
    activation_function = 'relu'

    execute_on = 'TIMESTEP_BEGIN'
  []
  [src_control_empty]
    type = LibtorchDRLControl
    parameters = "BCs/left_flux/value"
    observations = 'center_temp env_temp'

    input_timesteps = 2
    observation_scaling_factors = '0.03 0.03'
    observation_shift_factors = '270 270'
    action_scaling_factors = 100

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-8

  start_time = 0.0
  end_time = 18000
  dt = 1800.0
[]

[Outputs]
  csv = true
[]

(modules/stochastic_tools/test/tests/transfers/sampler_drl_control_transfer/sub.i)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 1
    xmin = 0.0
    xmax = 7.0
    nx = 3
  []
[]

[Variables]
  [temp]
    initial_condition = 300
  []
[]

[Kernels]
  [time]
    type = CoefTimeDerivative
    variable = temp
    Coefficient = '${fparse 1.00630182*1.225}'
  []
  [heat_conduc]
    type = MatDiffusion
    variable = temp
    diffusivity = 'k'
  []
[]

[BCs]
  [left_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'left'
    variable = temp
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = temp
    boundary = 'right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    expression = '15.0*sin(t/86400.0 *pi) + 273.0'
  []
  [design_function]
    type = ParsedFunction
    expression = '297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = 26.53832364
  []
[]

[Postprocessors]
  [center_temp_tend]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend'
  []
  [left_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [log_prob_left_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
[]

[Reporters]
  [T_reporter]
    type = AccumulateReporter
    reporters = 'center_temp_tend/value reward/value left_flux/value log_prob_left_flux/value'
  []
[]

[Controls]
  [src_control]
    type = LibtorchDRLControl
    parameters = "BCs/left_flux/value"
    observations = 'center_temp_tend'

    # Keep consistent with the trainer's timestep_window and input_timesteps.
    num_steps_in_period = 2
    input_timesteps = 2
    observation_scaling_factors = '0.03'
    observation_shift_factors = '270'
    action_scaling_factors = 100
    stochastic = true

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-8

  start_time = 0.0
  end_time = 86400
  dt = 14400.0
[]

[Outputs]
  console = false
[]

(modules/stochastic_tools/test/tests/transfers/libtorch_nn_transfer/libtorch_drl_control_sub.i)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 1
    xmin = 0.0
    xmax = 7.0
    nx = 3
  []
[]

[Variables]
  [temp]
    initial_condition = 300
  []
[]

[Kernels]
  [time]
    type = CoefTimeDerivative
    variable = temp
    Coefficient = '${fparse 1.00630182*1.225}'
  []
  [heat_conduc]
    type = MatDiffusion
    variable = temp
    diffusivity = 'k'
  []
[]

[BCs]
  [left_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'left'
    variable = temp
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = temp
    boundary = 'right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    expression = '15.0*sin(t/86400.0 *pi) + 273.0'
  []
  [design_function]
    type = ParsedFunction
    expression = '297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = 26.53832364
  []
[]

[Postprocessors]
  [center_temp]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [center_temp_tend]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [env_temp]
    type = FunctionValuePostprocessor
    function = temp_env
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend env_temp'
  []
  [left_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [log_prob_left_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
[]

[Reporters]
  [T_reporter]
    type = AccumulateReporter
    reporters = 'center_temp_tend/value env_temp/value reward/value left_flux/value log_prob_left_flux/value'
    outputs = 'csv_out'
  []
  [nn_parameters]
    type = LibtorchArtificialNeuralNetParameters
    control_name = src_control
    outputs = json_out
  []
[]

[Controls]
  [src_control]
    type = LibtorchDRLControl
    parameters = "BCs/left_flux/value"
    observations = 'center_temp env_temp'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 2
    observation_scaling_factors = '0.03 0.03'
    observation_shift_factors = '270 270'
    action_scaling_factors = 100

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-8

  start_time = 0.0
  end_time = 86400
  dt = 14400.0
[]

[Outputs]
  [json_out]
    type = JSON
    execute_on = FINAL
    execute_system_information_on = NONE
  []
[]

(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_sub.i)

air_density = 1.184 # kg/m3
air_cp = 1000 # J/(kg K)
air_effective_k = 0.5 # W/(m K)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 2
    xmin = 0.0
    xmax = 7.0
    ymin = 0.0
    ymax = 5.0
    nx = 20
    ny = 20
  []
[]

[Variables]
  [T]
    initial_condition = 297
  []
[]

[Kernels]
  [time_derivative]
    type = CoefTimeDerivative
    variable = T
    Coefficient = '${fparse air_density*air_cp}'
  []
  [heat_conduction]
    type = MatDiffusion
    variable = T
    diffusivity = 'k'
  []
[]

[BCs]
  [top_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'top'
    variable = T
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = T
    boundary = 'left right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    expression = '15.0*sin(t/86400.0*pi) + 273.0'
  []
  [design_function]
    type = ParsedFunction
    expression = '297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = ${air_effective_k}
  []
[]

[Postprocessors]
  [center_temp]
    type = PointValue
    variable = T
    point = '3.5 2.5 0.0'
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [center_temp_tend]
    type = PointValue
    variable = T
    point = '3.5 2.5 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [env_temp]
    type = FunctionValuePostprocessor
    function = temp_env
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend'
  []
  [top_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
  []
  [log_prob_top_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
  []
[]

[Reporters]
  [T_reporter]
    type = AccumulateReporter
    reporters = 'center_temp_tend/value reward/value top_flux/value log_prob_top_flux/value'
  []
[]

[Controls]
  [src_control]
    type = LibtorchDRLControl
    parameters = "BCs/top_flux/value"
    observations = 'center_temp_tend'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 1
    observation_scaling_factors = '0.03'
    observation_shift_factors = '290'
    action_scaling_factors = 20
    stochastic = true

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_hypre_type'
  petsc_options_value = 'hypre boomeramg'
  line_search = 'none'

  nl_rel_tol = 1e-6

  start_time = 0.0
  end_time = 86400
  dt = ${fparse 86400/80}
[]

[Outputs]
  console = false
[]

Overview
Execution Model
Input Parameters
Input Files