- control_nameController object name.
C++ Type:std::string
Controllable:No
Description:Controller object name.
- trainer_nameTrainer object that owns the latest controller network.
C++ Type:UserObjectName
Controllable:No
Description:Trainer object that owns the latest controller network.
SamplerDRLControlTransfer
Copies a DRL actor from a trainer object on the main app to a LibtorchDRLControl object on the subapp.
Overview
SamplerDRLControlTransfer copies the current actor network of a LibtorchDRLControlTrainer on the main application into a LibtorchDRLControl object on every sub-application of a sampler-driven MultiApp, such as SamplerFullSolveMultiApp. Each sub-application then runs one rollout of the latest policy, and the resulting trajectories are sent back to the trainer for the next policy update. "trainer_name" names the trainer on the main application, and "control_name" names the control in the sub-application input. After the network is copied, the transfer executes the control once, so that each rollout starts from an action of the current policy.
When the MultiApp runs in batch-reset or batch-restore mode, the transfer also reseeds the stochastic action sampling of the control for every sampler row. The seed is built from the trainer's "seed", the sampler row index, and the main-application time step. Every rollout of every training iteration thus draws its own action sequence, and that sequence does not depend on how the sampler rows are distributed among processors.
The transfer checks that the control executes the policy the trainer learns, and reports an error unless all of the following hold:
"timestep_window" equals "num_steps_in_period", so that each training transition corresponds to exactly one policy evaluation;
"input_timesteps" equals "input_timesteps", so that the control stacks the same observation history the actor was trained on;
the number of reporters in "observation" equals the number of postprocessors in "observations"; and
the number of reporters in "control" equals the number of controllable parameters in "parameters".
To transfer a trained network into a deterministic LibtorchNeuralNetControl instead, use LibtorchNeuralNetControlTransfer.
Example Input Syntax
In this example, the transfer copies the actor of the nn_trainer trainer into the src_control control of each runner sub-application before the sub-applications run. The dummy sampler has four rows, so every training iteration collects four rollouts in batch mode.
[Transfers<<<{"href": "../../../syntax/Transfers/index.html"}>>>]
[nn_transfer]
type = SamplerDRLControlTransfer<<<{"description": "Copies a DRL actor from a trainer object on the main app to a LibtorchDRLControl object on the subapp.", "href": "SamplerDRLControlTransfer.html"}>>>
to_multi_app<<<{"description": "The name of the MultiApp to transfer the data to"}>>> = runner
trainer_name<<<{"description": "Trainer object that owns the latest controller network."}>>> = nn_trainer
control_name<<<{"description": "Controller object name."}>>> = src_control
sampler<<<{"description": "A the Sampler object that Transfer is associated.."}>>> = dummy
[]
[]The control on the sub-application evaluates the policy every two time steps on a two-step observation history, matching the trainer's timestep_window = 2 and input_timesteps = 2.
[Controls<<<{"href": "../../../syntax/Controls/index.html"}>>>]
[src_control]
type = LibtorchDRLControl<<<{"description": "Sets the value of multiple 'Real' input parameters and postprocessors based on a Deep Reinforcement Learning (DRL) neural network trained using a PPO algorithm.", "href": "../controls/LibtorchDRLControl.html"}>>>
parameters<<<{"description": "Controllable input parameters driven by the network."}>>> = "BCs/left_flux/value"
observations<<<{"description": "Postprocessors used as the current observation vector."}>>> = 'center_temp_tend'
# Keep consistent with the trainer's timestep_window and input_timesteps.
num_steps_in_period<<<{"description": "Number of time steps between policy evaluations. The observation history advances once per period, so the trainer's 'timestep_window' should match this value."}>>> = 2
input_timesteps<<<{"description": "Number of recent timesteps to stack into each network input."}>>> = 2
observation_scaling_factors<<<{"description": "Optional multipliers applied after shifting the observation values."}>>> = '0.03'
observation_shift_factors<<<{"description": "Optional offsets applied to the observation values before scaling."}>>> = '270'
action_scaling_factors<<<{"description": "Per-action scaling embedded in the controller outputs so saved checkpoints stay in physical units."}>>> = 100
stochastic<<<{"description": "If true, sample from the policy distribution; otherwise use the deterministic action."}>>> = true
execute_on<<<{"description": "The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html."}>>> = 'TIMESTEP_BEGIN'
[]
[]Input Parameters
- check_multiapp_execute_onFalseWhen false the check between the multiapp and transfer execute on flags is not performed.
Default:False
C++ Type:bool
Controllable:No
Description:When false the check between the multiapp and transfer execute on flags is not performed.
- displaced_source_meshFalseWhether or not to use the displaced mesh for the source mesh.
Default:False
C++ Type:bool
Controllable:No
Description:Whether or not to use the displaced mesh for the source mesh.
- displaced_target_meshFalseWhether or not to use the displaced mesh for the target mesh.
Default:False
C++ Type:bool
Controllable:No
Description:Whether or not to use the displaced mesh for the target mesh.
- execute_after_from_multiappFalseWhether to execute the transfer after the from_multiapp has executed when both share an execute_on flag. Note that this setting is only used to move the execution of BETWEEN_MULTIAPP (siblings) transfers, FROM_ and TO_MULTIAPP transfers are always executed after the from_multiapp and before the to_multiapp respectively.
Default:False
C++ Type:bool
Controllable:No
Description:Whether to execute the transfer after the from_multiapp has executed when both share an execute_on flag. Note that this setting is only used to move the execution of BETWEEN_MULTIAPP (siblings) transfers, FROM_ and TO_MULTIAPP transfers are always executed after the from_multiapp and before the to_multiapp respectively.
- execute_onSAME_AS_MULTIAPPThe list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
Default:SAME_AS_MULTIAPP
C++ Type:ExecFlagEnum
Controllable:No
Description:The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
- samplerA the Sampler object that Transfer is associated..
C++ Type:SamplerName
Controllable:No
Description:A the Sampler object that Transfer is associated..
- to_multi_appThe name of the MultiApp to transfer the data to
C++ Type:MultiAppName
Controllable:No
Description:The name of the MultiApp to transfer the data to
Optional Parameters
- _called_legacy_paramsTrue
Default:True
C++ Type:bool
Controllable:No
- control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector<std::string>
Controllable:No
Description:Adds user-defined labels for accessing object parameters via control logic.
- enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Controllable:Yes
Description:Set the enabled status of the MooseObject.
- skip_coordinate_collapsingTrueWhether to skip coordinate collapsing (translation and rotation are still performed, only XYZ, RZ etc collapsing is skipped) when performing mapping and inverse mapping coordinate transformation operations. This parameter should only be set by users who really know what they're doing.
Default:True
C++ Type:bool
Controllable:No
Description:Whether to skip coordinate collapsing (translation and rotation are still performed, only XYZ, RZ etc collapsing is skipped) when performing mapping and inverse mapping coordinate transformation operations. This parameter should only be set by users who really know what they're doing.
- use_displaced_meshFalseWhether or not this object should use the displaced mesh for computation. Note that in the case this is true but no displacements are provided in the Mesh block the undisplaced mesh will still be used.
Default:False
C++ Type:bool
Controllable:No
Description:Whether or not this object should use the displaced mesh for computation. Note that in the case this is true but no displacements are provided in the Mesh block the undisplaced mesh will still be used.