ScaledAbsDifferenceDRLRewardFunction

Evaluates a scaled absolute difference reward function for a process which is controlled by a Deep Reinforcement Learning based surrogate.

Overview

Function describing the reward of for a Deep Reinforcement Learning algorithm in the form of:

var element = document.getElementById("moose-equation-f76bb15c-09ae-4954-8d29-e4a422982934");katex.render("r = C_1 |x_{target}-x_{current}| + C_2", element, {displayMode:true,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});

where $var element = document.getElementById("moose-equation-549215f1-c7f8-45f4-a12c-bb81eaa55617");katex.render("C_1", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ and $var element = document.getElementById("moose-equation-afdec20e-70f9-4ede-8bad-3943b11e50c0");katex.render("C_2", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ constants can be determined by the user. Furthermore, $var element = document.getElementById("moose-equation-d6469eba-951b-40d1-8337-9d9003622842");katex.render("x_{current}", element, {displayMode:false,throwOnError:false,macros:{"\\eqc":"\\,,","\\eqp":"\\,.","\\pd":"\\frac{\\partial #1}{\\partial #2}","\\pr":"\\left(#1\\right)","\\ddt":"\\frac{d #1}{d t}"}});$ is a measured data, typically supplied by a postprocessor. For an example on how to use it in a DRL setting, see LibtorchDRLControlTrainer.

Input Parameters

design_functionThe desired value to reach.
C++ Type:FunctionName
Controllable:No
Description:The desired value to reach.
observed_valueThe name of the Postprocessor that contains the observed value.
C++ Type:PostprocessorName
Controllable:No
Description:The name of the Postprocessor that contains the observed value.

Required Parameters

c1101st coefficient in the reward function.
Default:10
C++ Type:double
Controllable:No
Description:1st coefficient in the reward function.
c212nd coefficient in the reward function.
Default:1
C++ Type:double
Controllable:No
Description:2nd coefficient in the reward function.
execute_onLINEARThe list of flag(s) indicating when this object should be executed, the available options include FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, NONLINEAR, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, FINAL, CUSTOM, ALWAYS.
Default:LINEAR
C++ Type:ExecFlagEnum
Options:FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, NONLINEAR, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, FINAL, CUSTOM, ALWAYS
Controllable:No
Description:The list of flag(s) indicating when this object should be executed, the available options include FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, NONLINEAR, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, FINAL, CUSTOM, ALWAYS.

Optional Parameters

control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector<std::string>
Controllable:No
Description:Adds user-defined labels for accessing object parameters via control logic.
enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Controllable:No
Description:Set the enabled status of the MooseObject.

Advanced Parameters

Input Files

(modules/stochastic_tools/test/tests/functions/drl_reward/drl_reward.i)
(modules/stochastic_tools/test/tests/transfers/libtorch_nn_transfer/libtorch_drl_control_sub.i)
(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_sub.i)

(modules/stochastic_tools/test/tests/functions/drl_reward/drl_reward.i)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 1
    xmin = 0.0
    xmax = 7.0
    nx = 3
  []
[]

[Variables]
  [temp]
    initial_condition = 300
  []
[]

[Kernels]
  [time]
    type = CoefTimeDerivative
    variable = temp
    Coefficient = '${fparse 1.00630182*1.225}'
  []
  [heat_conduc]
    type = MatDiffusion
    variable = temp
    diffusivity = 'k'
  []
[]

[BCs]
  [dirichlet]
    type = FunctionDirichletBC
    function = "200"
    variable = temp
    boundary = 'right'
  []
[]

[Functions]
  [design_function]
    type = ParsedFunction
    value = 't/3600*297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = 26.53832364
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-8

  start_time = 0.0
  end_time = 3600
  dt = 1800
[]

[Postprocessors]
  [center_temp_tend]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend'
  []
[]

[Outputs]
  csv = true
[]

(modules/stochastic_tools/test/tests/transfers/libtorch_nn_transfer/libtorch_drl_control_sub.i)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 1
    xmin = 0.0
    xmax = 7.0
    nx = 3
  []
[]

[Variables]
  [temp]
    initial_condition = 300
  []
[]

[Kernels]
  [time]
    type = CoefTimeDerivative
    variable = temp
    Coefficient = '${fparse 1.00630182*1.225}'
  []
  [heat_conduc]
    type = MatDiffusion
    variable = temp
    diffusivity = 'k'
  []
[]

[BCs]
  [left_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'left'
    variable = temp
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = temp
    boundary = 'right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    value = '15.0*sin(t/86400.0 *pi) + 273.0'
  []
  [design_function]
    type = ParsedFunction
    value = '297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = 26.53832364
  []
[]

[Postprocessors]
  [center_temp]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [center_temp_tend]
    type = PointValue
    variable = temp
    point = '3.5 0.0 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [env_temp]
    type = FunctionValuePostprocessor
    function = temp_env
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend env_temp'
  []
  [left_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [log_prob_left_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
[]

[Reporters]
  [T_reporter]
    type = AccumulateReporter
    reporters = 'center_temp_tend/value env_temp/value reward/value left_flux/value log_prob_left_flux/value'
    outputs = 'csv_out'
  []
  [nn_parameters]
    type = LibtorchArtificialNeuralNetParameters
    control_name = src_control
    outputs = json_out
  []
[]

[Controls]
  [src_control]
    type = LibtorchDRLControl
    parameters = "BCs/left_flux/value"
    responses = 'center_temp env_temp'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 2
    response_scaling_factors = '0.03 0.03'
    response_shift_factors = '270 270'
    action_standard_deviations = '0.1'
    action_scaling_factors = 100

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-8

  start_time = 0.0
  end_time = 86400
  dt = 14400.0
[]

[Outputs]
  [json_out]
    type = JSON
    execute_on = FINAL
    execute_system_information_on = NONE
  []
[]

(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_sub.i)

air_density = 1.184 # kg/m3
air_cp = 1000 # J/(kg K)
air_effective_k = 0.5 # W/(m K)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 2
    xmin = 0.0
    xmax = 7.0
    ymin = 0.0
    ymax = 5.0
    nx = 35
    ny = 25
  []
[]

[Variables]
  [T]
    initial_condition = 297
  []
[]

[Kernels]
  [time_derivative]
    type = CoefTimeDerivative
    variable = T
    Coefficient = '${fparse air_density*air_cp}'
  []
  [heat_conduction]
    type = MatDiffusion
    variable = T
    diffusivity = 'k'
  []
[]

[BCs]
  [top_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'top'
    variable = T
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = T
    boundary = 'left right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    value = '15.0*sin(t/86400.0*pi) + 273.0'
  []
  [design_function]
    type = ParsedFunction
    value = '297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = ${air_effective_k}
  []
[]

[Postprocessors]
  [center_temp]
    type = PointValue
    variable = T
    point = '3.5 2.5 0.0'
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [center_temp_tend]
    type = PointValue
    variable = T
    point = '3.5 2.5 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [env_temp]
    type = FunctionValuePostprocessor
    function = temp_env
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend env_temp'
  []
  [top_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
  []
  [log_prob_top_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
  []
[]

[Reporters]
  [T_reporter]
    type = AccumulateReporter
    reporters = 'center_temp_tend/value env_temp/value reward/value top_flux/value log_prob_top_flux/value'
  []
[]

[Controls]
  inactive = 'src_control_final'
  [src_control]
    type = LibtorchDRLControl
    parameters = "BCs/top_flux/value"
    responses = 'center_temp_tend env_temp'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 2
    response_scaling_factors = '0.03 0.03'
    response_shift_factors = '290 290'
    action_standard_deviations = '0.02'
    action_scaling_factors = 200

    execute_on = 'TIMESTEP_BEGIN'
  []
  [src_control_final]
    type = LibtorchNeuralNetControl

    filename = 'mynet_control.net'
    num_neurons_per_layer = '16 6'
    activation_function = 'relu'

    parameters = "BCs/top_flux/value"
    responses = 'center_temp_tend env_temp'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 2
    response_scaling_factors = '0.03 0.03'
    response_shift_factors = '290 290'
    action_standard_deviations = '0.02'
    action_scaling_factors = 200

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-7

  start_time = 0.0
  end_time = 86400
  dt = 900.0
[]

[Outputs]
  console = false
  [c]
    type = CSV
    execute_on = FINAL
  []
[]

Overview
Input Parameters
Input Files