LibtorchDRLControlTrainer

Trains a neural network controller using fixed-horizon PPO on top of the libtorch RL core.

Overview

This object trains an on-policy actor-critic controller using Proximal Policy Optimization (PPO) Schulman et al. (2017) together with generalized advantage estimation (GAE) Schulman et al. (2015). Reporter trajectories are assembled into transitions, flattened across samples, shuffled into mini-batches, and then used to update separate actor and critic neural networks.

Algorithm Summary

For input_timesteps = H, the observation passed to the actor and critic is the stacked history $var element = document.getElementById("moose-equation-fd64b3ea-e8cc-4aab-891d-91ac0f62943d");katex.render("o_t = \\left[s_t,\\ s_{t-1},\\ \\ldots,\\ s_{t-H+1}\\right],", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ where early missing history entries are filled with the earliest available reporter value.

Each collected trajectory is converted into the tuples $var element = document.getElementById("moose-equation-92d72330-b5ed-4b91-8c55-ef4e267ad3be");katex.render("(o_t, a_t, \\log \\pi_{\\mathrm{old}}(a_t|o_t), r_t, o_{t+1})", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ . The "shift_outputs" option aligns the reported action and log-probability sequences with the reward sequence when the control is applied at the beginning of a time step and the reward is measured at the end of that step. The "timestep_window" option can be used to downsample long reporter trajectories before these transitions are assembled. For timestep_window = W, the trainer keeps every $var element = document.getElementById("moose-equation-cf8ea51a-e9ad-4e2b-b30b-b56582157b09");katex.render("W", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ -th reporter entry, so consecutive entries of the stacked history $var element = document.getElementById("moose-equation-e3e29ef9-c6ac-4998-8035-8b9225199082");katex.render("o_t", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ are $var element = document.getElementById("moose-equation-099c8abe-8957-45b9-81bb-90197a3d7710");katex.render("W", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ time steps apart. $var element = document.getElementById("moose-equation-aceaec4c-6f56-4c3c-96bc-6f17974f5eeb");katex.render("W", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ should match "num_steps_in_period" on the controller, which advances its observation history once per control period.

The critic is first evaluated on $var element = document.getElementById("moose-equation-121ce9e1-4201-4041-8392-a2fa31973b9b");katex.render("o_t", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ and $var element = document.getElementById("moose-equation-227c0cfa-79f6-4e75-bcc5-2f07f22f434f");katex.render("o_{t+1}", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ , and the temporal-difference residual is computed as $var element = document.getElementById("moose-equation-ea4c9e8d-b889-4fe8-af54-b1cd680519fd");katex.render("\\delta_t = r_t + \\gamma V_{\\phi}(o_{t+1}) - V_{\\phi}(o_t).", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ The trainer then computes the reverse-time GAE recursion $var element = document.getElementById("moose-equation-da164b10-cf00-4571-b218-bf549f64ba9e");katex.render("\\hat{A}_t = \\delta_t + \\gamma \\lambda \\hat{A}_{t+1},", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ with critic regression targets $var element = document.getElementById("moose-equation-add8b535-9340-4f88-85fb-5168b910f997");katex.render("\\hat{V}_t = \\hat{A}_t + V_{\\phi}(o_t).", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$

For each mini-batch, PPO uses the probability ratio $var element = document.getElementById("moose-equation-00d20417-2a87-4f0b-88a7-e3c7c843670f");katex.render("r_t(\\theta) = \\exp\\left(\\log \\pi_{\\theta}(a_t|o_t) - \\log \\pi_{\\mathrm{old}}(a_t|o_t)\\right),", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ and the actor objective implemented here is $var element = document.getElementById("moose-equation-a9504340-0ff3-4182-a0b0-c0c0f378e5a2");katex.render("L_{\\mathrm{actor}} = -\\frac{1}{N}\\sum_t \\left[ \\min\\left(r_t(\\theta)\\hat{A}_t, \\mathrm{clip}\\left(r_t(\\theta), 1-\\epsilon, 1+\\epsilon\\right)\\hat{A}_t\\right) + c_H \\mathcal{H}\\left[\\pi_{\\theta}(\\cdot|o_t)\\right] \\right].", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ The critic is trained with a mean-squared-error loss, $var element = document.getElementById("moose-equation-250abab1-3230-4d3e-916a-25aa1e472057");katex.render("L_{\\mathrm{critic}} = \\frac{1}{N}\\sum_t \\left(V_{\\phi}(o_t) - \\hat{V}_t\\right)^2.", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$

If "standardize_advantage" is set, each sampled mini-batch uses zero-mean, unit-variance advantages before the PPO loss is evaluated.

Notes

The actor embeds the configured input shifting, input scaling, and action scaling directly into the neural-network module so that transferred and checkpointed controllers operate in the same normalized coordinates used during training.

The trainer updates the actor and critic with separate Adam optimizers and then broadcasts the updated parameters so all MPI ranks hold the same networks after each PPO update.

Example Input File Syntax

Input Parameters

controlReporters containing the values of the controlled quantities (control signals) from the model simulations.
C++ Type:std::vector<ReporterName>
Controllable:No
Description:Reporters containing the values of the controlled quantities (control signals) from the model simulations.
control_learning_rateLearning rate used by the actor optimizer.
C++ Type:Real
Unit:(no unit assumed)
Range:0
Controllable:No
Description:Learning rate used by the actor optimizer.
critic_learning_rateLearning rate used by the critic optimizer.
C++ Type:Real
Unit:(no unit assumed)
Range:0
Controllable:No
Description:Learning rate used by the critic optimizer.
log_probabilityReporters containing the log probabilities of the actions taken during the simulations.
C++ Type:std::vector<ReporterName>
Controllable:No
Description:Reporters containing the log probabilities of the actions taken during the simulations.
num_control_neurons_per_layerNumber of neurons per layer for the control neural network.
C++ Type:std::vector<unsigned int>
Controllable:No
Description:Number of neurons per layer for the control neural network.
num_critic_neurons_per_layerHidden-layer widths for the critic network.
C++ Type:std::vector<unsigned int>
Controllable:No
Description:Hidden-layer widths for the critic network.
num_epochsNumber of epochs for the training.
C++ Type:unsigned int
Controllable:No
Description:Number of epochs for the training.
observationReporter values containing the observation values from the model.
C++ Type:std::vector<ReporterName>
Controllable:No
Description:Reporter values containing the observation values from the model.
rewardReporter containing the earned time-dependent rewards from the simulation.
C++ Type:ReporterName
Controllable:No
Description:Reporter containing the earned time-dependent rewards from the simulation.

Required Parameters

action_scaling_factorsScale factors embedded into the trained policy outputs so transferred and checkpointed controllers operate in physical units.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Scale factors embedded into the trained policy outputs so transferred and checkpointed controllers operate in physical units.
average_reward_over_timestep_windowFalseAverage reward reporter values over each timestep_window interval instead of sampling one value at the end of the interval.
Default:False
C++ Type:bool
Controllable:No
Description:Average reward reporter values over each timestep_window interval instead of sampling one value at the end of the interval.
batch_size100Number of flattened samples per mini-batch.
Default:100
C++ Type:unsigned int
Controllable:No
Description:Number of flattened samples per mini-batch.
clip_parameter0.2Clip parameter used while clamping the advantage value.
Default:0.2
C++ Type:Real
Unit:(no unit assumed)
Controllable:No
Description:Clip parameter used while clamping the advantage value.
control_activation_functionsrelu Activation name for each actor hidden layer, or one shared value for all layers.
Default:relu
C++ Type:std::vector<std::string>
Controllable:No
Description:Activation name for each actor hidden layer, or one shared value for all layers.
critic_activation_functionsrelu Activation name for each critic hidden layer, or one shared value for all layers.
Default:relu
C++ Type:std::vector<std::string>
Controllable:No
Description:Activation name for each critic hidden layer, or one shared value for all layers.
decay_factor1Discount factor used when building PPO return and GAE targets.
Default:1
C++ Type:Real
Unit:(no unit assumed)
Range:0.0<=decay_factor<=1.0
Controllable:No
Description:Discount factor used when building PPO return and GAE targets.
entropy_coeff0.01Entropy bonus coefficient used in the PPO actor loss.
Default:0.01
C++ Type:Real
Unit:(no unit assumed)
Controllable:No
Description:Entropy bonus coefficient used in the PPO actor loss.
filenameThe name of the file which will be associated with the saved/loaded data.
C++ Type:FileName
Controllable:No
Description:The name of the file which will be associated with the saved/loaded data.
filename_baseBase filename used when writing actor and critic checkpoints.
C++ Type:std::string
Controllable:No
Description:Base filename used when writing actor and critic checkpoints.
input_timesteps1Number of time steps to use in the input data, if larger than 1, data from the previous timesteps will be used as inputs in the training.
Default:1
C++ Type:unsigned int
Range:1<=input_timesteps
Controllable:No
Description:Number of time steps to use in the input data, if larger than 1, data from the previous timesteps will be used as inputs in the training.
lambda_factor1GAE lambda.
Default:1
C++ Type:Real
Unit:(no unit assumed)
Range:0.0<=lambda_factor<=1.0
Controllable:No
Description:GAE lambda.
loss_print_frequency0Print PPO loss values every N updates. Use 0 to stay quiet.
Default:0
C++ Type:unsigned int
Controllable:No
Description:Print PPO loss values every N updates. Use 0 to stay quiet.
max_control_valueOptional upper bounds for each control signal.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional upper bounds for each control signal.
min_control_valueOptional lower bounds for each control signal.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional lower bounds for each control signal.
observation_scaling_factorsOptional multipliers applied after shifting the observed state values.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional multipliers applied after shifting the observed state values.
observation_shift_factorsOptional offsets applied to the observed state values before scaling.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional offsets applied to the observed state values before scaling.
read_from_fileFalseSwitch to read the neural network parameters from a file.
Default:False
C++ Type:bool
Controllable:No
Description:Switch to read the neural network parameters from a file.
seed11Random number generator seed for stochastic optimizers.
Default:11
C++ Type:unsigned int
Controllable:No
Description:Random number generator seed for stochastic optimizers.
shift_outputsTrueWhether to shift rollout outputs so observations and actions line up in time.
Default:True
C++ Type:bool
Controllable:No
Description:Whether to shift rollout outputs so observations and actions line up in time.
standardize_advantageTrueSwitch to enable the shifting and normalization of the advantages in the PPO algorithm.
Default:True
C++ Type:bool
Controllable:No
Description:Switch to enable the shifting and normalization of the advantages in the PPO algorithm.
state_independent_stdTrueIf true, learn one Gaussian log-std parameter per action dimension. If false, learn a state-dependent std head as in the older MOOSE actor implementation.
Default:True
C++ Type:bool
Controllable:No
Description:If true, learn one Gaussian log-std parameter per action dimension. If false, learn a state-dependent std head as in the older MOOSE actor implementation.
timestep_window1Use every nth reporter timestep when assembling trajectories. Observation history lags also span this many timesteps, so it should match the controller's 'num_steps_in_period'.
Default:1
C++ Type:unsigned int
Range:1<=timestep_window
Controllable:No
Description:Use every nth reporter timestep when assembling trajectories. Observation history lags also span this many timesteps, so it should match the controller's 'num_steps_in_period'.
update_frequency1Number of transient simulation data to collect for updating the controller neural network.
Default:1
C++ Type:unsigned int
Range:1<=update_frequency
Controllable:No
Description:Number of transient simulation data to collect for updating the controller neural network.

Optional Parameters

allow_duplicate_execution_on_initialFalseIn the case where this UserObject is depended upon by an initial condition, allow it to be executed twice during the initial setup (once before the IC and again after mesh adaptivity (if applicable).
Default:False
C++ Type:bool
Controllable:No
Description:In the case where this UserObject is depended upon by an initial condition, allow it to be executed twice during the initial setup (once before the IC and again after mesh adaptivity (if applicable).
execute_onTIMESTEP_ENDThe list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
Default:TIMESTEP_END
C++ Type:ExecFlagEnum
Options:XFEM_MARK, FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, LINEAR_CONVERGENCE, NONLINEAR, NONLINEAR_CONVERGENCE, POSTCHECK, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, MULTIAPP_FIXED_POINT_CONVERGENCE, MULTISYSTEM_FIXED_POINT_ITERATION_END, FINAL, CUSTOM, TRANSFER
Controllable:No
Description:The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
execution_order_group0Execution order groups are executed in increasing order (e.g., the lowest number is executed first). Note that negative group numbers may be used to execute groups before the default (0) group. Please refer to the user object documentation for ordering of user object execution within a group.
Default:0
C++ Type:int
Controllable:No
Description:Execution order groups are executed in increasing order (e.g., the lowest number is executed first). Note that negative group numbers may be used to execute groups before the default (0) group. Please refer to the user object documentation for ordering of user object execution within a group.
force_postauxFalseForces the UserObject to be executed in POSTAUX
Default:False
C++ Type:bool
Controllable:No
Description:Forces the UserObject to be executed in POSTAUX
force_preauxFalseForces the UserObject to be executed in PREAUX
Default:False
C++ Type:bool
Controllable:No
Description:Forces the UserObject to be executed in PREAUX
force_preicFalseForces the UserObject to be executed in PREIC during initial setup
Default:False
C++ Type:bool
Controllable:No
Description:Forces the UserObject to be executed in PREIC during initial setup

Execution Scheduling Parameters

control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector<std::string>
Controllable:No
Description:Adds user-defined labels for accessing object parameters via control logic.
enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Controllable:Yes
Description:Set the enabled status of the MooseObject.
use_displaced_meshFalseWhether or not this object should use the displaced mesh for computation. Note that in the case this is true but no displacements are provided in the Mesh block the undisplaced mesh will still be used.
Default:False
C++ Type:bool
Controllable:No
Description:Whether or not this object should use the displaced mesh for computation. Note that in the case this is true but no displacements are provided in the Mesh block the undisplaced mesh will still be used.

Advanced Parameters

prop_getter_suffixAn optional suffix parameter that can be appended to any attempt to retrieve/get material properties. The suffix will be prepended with a '_' character.
C++ Type:MaterialPropertyName
Unit:(no unit assumed)
Controllable:No
Description:An optional suffix parameter that can be appended to any attempt to retrieve/get material properties. The suffix will be prepended with a '_' character.
use_interpolated_stateFalseFor the old and older state use projected material properties interpolated at the quadrature points. To set up projection use the ProjectedStatefulMaterialStorageAction.
Default:False
C++ Type:bool
Controllable:No
Description:For the old and older state use projected material properties interpolated at the quadrature points. To set up projection use the ProjectedStatefulMaterialStorageAction.

Material Property Retrieval Parameters

Input Files

(modules/stochastic_tools/test/tests/transfers/libtorch_nn_transfer/libtorch_drl_control_trainer.i)
(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_trainer.i)
(modules/stochastic_tools/test/tests/transfers/sampler_drl_control_transfer/trainer.i)
(modules/stochastic_tools/test/tests/surrogates/libtorch_drl/trainer_errors.i)

References

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.

BibTeX

@article{schulman2015gae,
    author = "Schulman, John and Moritz, Philipp and Levine, Sergey and Jordan, Michael and Abbeel, Pieter",
    title = "High-dimensional continuous control using generalized advantage estimation",
    journal = "arXiv preprint arXiv:1506.02438",
    year = "2015"
}

RIS

TY  - JOUR
AU  - Schulman, John
AU  - Moritz, Philipp
AU  - Levine, Sergey
AU  - Jordan, Michael
AU  - Abbeel, Pieter
TI  - High-dimensional continuous control using generalized advantage estimation
JO  - arXiv preprint arXiv:1506.02438
PY  - 2015
ER  -

Plain Text

John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

BibTeX

@article{schulman2017proximal,
    author = "Schulman, John and Wolski, Filip and Dhariwal, Prafulla and Radford, Alec and Klimov, Oleg",
    title = "Proximal policy optimization algorithms",
    journal = "arXiv preprint arXiv:1707.06347",
    year = "2017"
}

RIS

TY  - JOUR
AU  - Schulman, John
AU  - Wolski, Filip
AU  - Dhariwal, Prafulla
AU  - Radford, Alec
AU  - Klimov, Oleg
TI  - Proximal policy optimization algorithms
JO  - arXiv preprint arXiv:1707.06347
PY  - 2017
ER  -

Plain Text

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.

(modules/stochastic_tools/test/tests/transfers/libtorch_nn_transfer/libtorch_drl_control_trainer.i)

[StochasticTools]
[]

[Samplers]
  [dummy]
    type = CartesianProduct
    linear_space_items = '0 0.01 1'
  []
[]

[MultiApps]
  [runner]
    type = SamplerFullSolveMultiApp
    sampler = dummy
    input_files = 'libtorch_drl_control_sub.i'
  []
[]

[Transfers]
  [nn_transfer]
    type = LibtorchNeuralNetControlTransfer
    to_multi_app = runner
    trainer_name = nn_trainer
    control_name = src_control
  []
  [r_transfer]
    type = SamplerReporterTransfer
    from_multi_app = runner
    sampler = dummy
    stochastic_reporter = storage
    from_reporter = 'T_reporter/center_temp_tend:value T_reporter/env_temp:value T_reporter/reward:value T_reporter/left_flux:value T_reporter/log_prob_left_flux:value'
  []
[]

[Trainers]
  [nn_trainer]
    type = LibtorchDRLControlTrainer
    observation = 'storage/r_transfer:T_reporter:center_temp_tend:value storage/r_transfer:T_reporter:env_temp:value'
    control = 'storage/r_transfer:T_reporter:left_flux:value'
    log_probability = 'storage/r_transfer:T_reporter:log_prob_left_flux:value'
    reward = 'storage/r_transfer:T_reporter:reward:value'

    num_epochs = 10
    update_frequency = 2
    decay_factor = 0.0

    loss_print_frequency = 3

    critic_learning_rate = 0.0005
    num_critic_neurons_per_layer = '4 2'

    control_learning_rate = 0.0005
    num_control_neurons_per_layer = '4 2'

    # keep consistent with LibtorchNeuralNetControl
    input_timesteps = 2
    observation_scaling_factors = '0.03 0.03'
    observation_shift_factors = '270 270'
    action_scaling_factors = 100

    read_from_file = false
    shift_outputs = false
  []
[]

[Reporters]
  [storage]
    type = StochasticReporter
    parallel_type = ROOT
    outputs = none
  []
  [nn_parameters]
    type = DRLControlNeuralNetParameters
    trainer_name = nn_trainer
    outputs = json_out
  []
[]

[Executioner]
  type = Transient
  num_steps = 1
[]

[Outputs]
  file_base = train_out
  [json_out]
    type = JSON
    execute_on = TIMESTEP_BEGIN
    execute_system_information_on = NONE
  []
[]

(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_trainer.i)

[StochasticTools]
[]

[Samplers]
  [dummy]
    type = CartesianProduct
    linear_space_items = '0 0.01 1'
  []
[]

[MultiApps]
  [runner]
    type = SamplerFullSolveMultiApp
    sampler = dummy
    input_files = 'libtorch_drl_control_sub.i'
    mode = batch-reset
  []
[]

[Transfers]
  [nn_transfer]
    type = SamplerDRLControlTransfer
    to_multi_app = runner
    trainer_name = nn_trainer
    control_name = src_control
    sampler = dummy
  []
  [r_transfer]
    type = SamplerReporterTransfer
    from_multi_app = runner
    sampler = dummy
    stochastic_reporter = storage
    from_reporter = 'T_reporter/center_temp_tend:value T_reporter/reward:value T_reporter/top_flux:value T_reporter/log_prob_top_flux:value'
  []
[]

[Trainers]
  [nn_trainer]
    type = LibtorchDRLControlTrainer
    observation = 'storage/r_transfer:T_reporter:center_temp_tend:value'
    control = 'storage/r_transfer:T_reporter:top_flux:value'
    log_probability = 'storage/r_transfer:T_reporter:log_prob_top_flux:value'
    reward = 'storage/r_transfer:T_reporter:reward:value'

    num_epochs = 50
    update_frequency = 1
    decay_factor = 0.8
    lambda_factor = 1.0

    loss_print_frequency = 10

    critic_learning_rate = 0.001
    num_critic_neurons_per_layer = '32 16'
    critic_activation_functions = 'relu relu'

    control_learning_rate = 0.001
    num_control_neurons_per_layer = '32 16'
    control_activation_functions = 'relu relu'


    # keep consistent with LibtorchNeuralNetControl
    input_timesteps = 1
    observation_scaling_factors = '0.03'
    observation_shift_factors = '290'
    action_scaling_factors = 20

    standardize_advantage = true

    batch_size = 80

    read_from_file = false

    entropy_coeff = 0.0

    # min_control_value = ${fparse -0.1}
    # max_control_value = ${fparse 0.1}

  []
[]

[Reporters]
  [storage]
    type = StochasticReporter
    parallel_type = ROOT
    outputs = none
  []
  [reward]
    type = DRLRewardReporter
    drl_trainer_name = nn_trainer
  []
[]

[Executioner]
  type = Transient
  num_steps = 200
[]

[Outputs]
  file_base = output/train_out
  json = true
  time_step_interval = 1
  execute_on = TIMESTEP_END
[]

(modules/stochastic_tools/test/tests/transfers/sampler_drl_control_transfer/trainer.i)

[StochasticTools]
[]

[Samplers]
  [dummy]
    type = CartesianProduct
    linear_space_items = '0 1 4'
  []
[]

[MultiApps]
  [runner]
    type = SamplerFullSolveMultiApp
    sampler = dummy
    input_files = 'sub.i'
    mode = batch-reset
  []
[]

[Transfers]
  [nn_transfer]
    type = SamplerDRLControlTransfer
    to_multi_app = runner
    trainer_name = nn_trainer
    control_name = src_control
    sampler = dummy
  []
  [r_transfer]
    type = SamplerReporterTransfer
    from_multi_app = runner
    sampler = dummy
    stochastic_reporter = storage
    from_reporter = 'T_reporter/center_temp_tend:value T_reporter/reward:value T_reporter/left_flux:value T_reporter/log_prob_left_flux:value'
  []
[]

[Trainers]
  [nn_trainer]
    type = LibtorchDRLControlTrainer
    observation = 'storage/r_transfer:T_reporter:center_temp_tend:value'
    control = 'storage/r_transfer:T_reporter:left_flux:value'
    log_probability = 'storage/r_transfer:T_reporter:log_prob_left_flux:value'
    reward = 'storage/r_transfer:T_reporter:reward:value'

    num_epochs = 5
    update_frequency = 2
    decay_factor = 0.9
    batch_size = 8

    critic_learning_rate = 0.001
    num_critic_neurons_per_layer = '4'

    control_learning_rate = 0.001
    num_control_neurons_per_layer = '4'

    # Keep consistent with the controller's num_steps_in_period and input_timesteps.
    timestep_window = 2
    input_timesteps = 2
    observation_scaling_factors = '0.03'
    observation_shift_factors = '270'
    action_scaling_factors = 100

    # Bounded controls use the Beta actor.
    min_control_value = 0
    max_control_value = 2

    filename_base = 'sampler_drl'
  []
[]

[Reporters]
  [storage]
    type = StochasticReporter
    parallel_type = ROOT
    outputs = none
  []
  [reward]
    type = DRLRewardReporter
    drl_trainer_name = nn_trainer
  []
  [nn_parameters]
    type = DRLControlNeuralNetParameters
    trainer_name = nn_trainer
  []
[]

[Executioner]
  type = Transient
  num_steps = 4
[]

[Outputs]
  [json_out]
    type = JSON
    execute_system_information_on = NONE
  []
[]

(modules/stochastic_tools/test/tests/surrogates/libtorch_drl/trainer_errors.i)

[StochasticTools]
[]

[Reporters]
  # Two rollout samples with three raw entries each: the initial value and two time steps.
  [data]
    type = ConstantReporter
    real_vector_vector_names = 'observation action log_probability reward'
    real_vector_vector_values = '300 301 302; 300 302 304 | 0 1 2; 0 2 4 | 0 -1 -1; 0 -2 -2 | 0 1 2; 0 2 4'
  []
[]

[Trainers]
  [nn_trainer]
    type = LibtorchDRLControlTrainer
    observation = 'data/observation'
    control = 'data/action'
    log_probability = 'data/log_probability'
    reward = 'data/reward'

    num_epochs = 1
    critic_learning_rate = 0.001
    num_critic_neurons_per_layer = '2'
    control_learning_rate = 0.001
    num_control_neurons_per_layer = '2'

    filename_base = 'trainer_errors'
  []
[]

[Executioner]
  type = Transient
  num_steps = 1
[]

Overview
Algorithm Summary
Notes
Example Input File Syntax
Input Parameters
Input Files
References