LibtorchNeuralNetControl

Controls the value of multiple controllable input parameters using a Libtorch-based neural network.

Overview

This object controls a physical process using a neural network. The neural network can be either read from a file of transferred from a trainer. One can use the "filename" parameter to load the neural network from file. This capability supports two formats:

Reading entire neural networks using a TorchScript format in which case the user needs to specify "torch_script_format" parameter;
Reading neural network parameters from a binary (usually .net) file. In this case only parameters are read meaning that the structure of the neural network is not established in advance. For this reason, we need to specify the architecture using "num_neurons_per_layer" and "activation_function" parameters.

Input Parameters

parametersThe input parameter(s) to control.
C++ Type:std::vector<std::string>
Controllable:No
Description:The input parameter(s) to control.
responsesThe responses (prostprocessors) which are used for the control.
C++ Type:std::vector<PostprocessorName>
Unit:(no unit assumed)
Controllable:No
Description:The responses (prostprocessors) which are used for the control.

Required Parameters

action_scaling_factorsScale factor that multiplies the NN output to obtain a physically meaningful value.
C++ Type:std::vector<double>
Unit:(no unit assumed)
Controllable:No
Description:Scale factor that multiplies the NN output to obtain a physically meaningful value.
activation_functionrelu The type of activation functions to use. It is either one value or one value per hidden layer.
Default:relu
C++ Type:std::vector<std::string>
Controllable:No
Description:The type of activation functions to use. It is either one value or one value per hidden layer.
depends_onThe Controls that this control relies upon (i.e. must execute before this one)
C++ Type:std::vector<std::string>
Controllable:No
Description:The Controls that this control relies upon (i.e. must execute before this one)
execute_onINITIAL TIMESTEP_ENDThe list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
Default:INITIAL TIMESTEP_END
C++ Type:ExecFlagEnum
Options:XFEM_MARK, FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, NONLINEAR_CONVERGENCE, NONLINEAR, POSTCHECK, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, FINAL, CUSTOM, PRE_MULTIAPP_SETUP
Controllable:No
Description:The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
filenameDefine if the neural net is supposed to be loaded from a file.
C++ Type:std::string
Controllable:No
Description:Define if the neural net is supposed to be loaded from a file.
input_timesteps1Number of time steps to use in the input data, if larger than 1, data from the previous timesteps will be used as well as inputs in the training.
Default:1
C++ Type:unsigned int
Controllable:No
Description:Number of time steps to use in the input data, if larger than 1, data from the previous timesteps will be used as well as inputs in the training.
num_neurons_per_layerThe number of neurons on each hidden layer.
C++ Type:std::vector<unsigned int>
Controllable:No
Description:The number of neurons on each hidden layer.
response_scaling_factorsConstants which will be used to multiply the shifted response values. This is used for the manipulation of the neural net inputs for better training efficiency.
C++ Type:std::vector<double>
Unit:(no unit assumed)
Controllable:No
Description:Constants which will be used to multiply the shifted response values. This is used for the manipulation of the neural net inputs for better training efficiency.
response_shift_factorsConstants which will be used to shift the response values. This is used for the manipulation of the neural net inputs for better training efficiency.
C++ Type:std::vector<double>
Unit:(no unit assumed)
Controllable:No
Description:Constants which will be used to shift the response values. This is used for the manipulation of the neural net inputs for better training efficiency.
torch_script_formatFalseIf we want to load the neural net using the torch-script format.
Default:False
C++ Type:bool
Controllable:No
Description:If we want to load the neural net using the torch-script format.

Optional Parameters

control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector<std::string>
Controllable:No
Description:Adds user-defined labels for accessing object parameters via control logic.
enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Controllable:No
Description:Set the enabled status of the MooseObject.
implicitTrueDetermines whether this object is calculated using an implicit or explicit form
Default:True
C++ Type:bool
Controllable:No
Description:Determines whether this object is calculated using an implicit or explicit form

Advanced Parameters

Input Files

(test/tests/controls/libtorch_nn_control/read_control.i)
(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_sub.i)

Child Objects

(modules/stochastic_tools/include/libtorch/controls/LibtorchDRLControl.h)

warning

The detailed documentation of this object is only available when Moose is compiled with Libtorch. For instructions on how to compile Moose with Libtorch, visit the general installation webpage or click here.

(test/tests/controls/libtorch_nn_control/read_control.i)

pi = 3.14159265359
period = 0.25
diff_coeff = 0.5
cp = 1.0

[Functions]
  [src_func]
    type = ParsedFunction
    value = "sin(${pi}/${period}*t)"
  []
[]

[Mesh]
  [msh]
    type = GeneratedMeshGenerator
    dim = 2
    nx = 20
    xmin = -0.5
    xmax = 0.5
    ny = 20
    ymin = -0.5
    ymax = 0.5
  []
  [source_domain]
    type = ParsedSubdomainMeshGenerator
    input = msh
    combinatorial_geometry = '(x<0.2 & x>-0.2) & (y<0.2 & y>-0.2)'
    block_id = 1
  []
[]

[Variables]
  [T]
    initial_condition = 1
  []
[]

[Kernels]
  [diffusion]
    type = CoefDiffusion
    variable = T
    coef = ${diff_coeff}
  []
  [source]
    type = BodyForce
    variable = T
    function = src_func
    block = 1
  []
  [anti_source]
    type = BodyForce
    variable = T
    value = 0
    block = 1
  []
  [time_deriv]
    type = CoefTimeDerivative
    Coefficient = ${cp}
    variable = T
  []
[]

[BCs]
  [neumann_rest]
    type = NeumannBC
    variable = T
    boundary = 'left right top bottom'
    value = 0
  []
[]

[Executioner]
  type = Transient
  num_steps = 25
  dt = 0.1
  solve_type = NEWTON
  petsc_options_iname = '-pc_type'
  petsc_options_value = 'lu'
  nl_abs_tol = 1e-12
  line_search = 'none'
[]

[Postprocessors]
  [T_max]
    type = NodalExtremeValue
    variable = T
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [control_value]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
[]

[Controls]
  [src_control]
    type = LibtorchNeuralNetControl
    parameters = "Kernels/anti_source/value"
    responses = 'T_max'
    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Reporters]
  inactive = 'nn_parameters'
  [T_reporter]
    type = AccumulateReporter
    reporters = 'T_max/value control_value/value'
    outputs = csv_out
  []
  [nn_parameters]
    type = LibtorchArtificialNeuralNetParameters
    control_name = src_control
    execute_on = FINAL
    outputs = json_out
  []
[]

[Outputs]
  [csv_out]
    type = CSV
    execute_on = FINAL
  []
  [json_out]
    type = JSON
    execute_on = FINAL
    execute_system_information_on = NONE
  []
[]

(modules/stochastic_tools/examples/libtorch_drl_control/libtorch_drl_control_sub.i)

air_density = 1.184 # kg/m3
air_cp = 1000 # J/(kg K)
air_effective_k = 0.5 # W/(m K)

[Mesh]
  [mesh]
    type = GeneratedMeshGenerator
    dim = 2
    xmin = 0.0
    xmax = 7.0
    ymin = 0.0
    ymax = 5.0
    nx = 35
    ny = 25
  []
[]

[Variables]
  [T]
    initial_condition = 297
  []
[]

[Kernels]
  [time_derivative]
    type = CoefTimeDerivative
    variable = T
    Coefficient = '${fparse air_density*air_cp}'
  []
  [heat_conduction]
    type = MatDiffusion
    variable = T
    diffusivity = 'k'
  []
[]

[BCs]
  [top_flux]
    type = NeumannBC
    value = 0.0
    boundary = 'top'
    variable = T
  []
  [dirichlet]
    type = FunctionDirichletBC
    function = temp_env
    variable = T
    boundary = 'left right'
  []
[]

[Functions]
  [temp_env]
    type = ParsedFunction
    value = '15.0*sin(t/86400.0*pi) + 273.0'
  []
  [design_function]
    type = ParsedFunction
    value = '297'
  []
  [reward_function]
    type = ScaledAbsDifferenceDRLRewardFunction
    design_function = design_function
    observed_value = center_temp_tend
    c1 = 1
    c2 = 10
  []
[]

[Materials]
  [constant]
    type = GenericConstantMaterial
    prop_names = 'k'
    prop_values = ${air_effective_k}
  []
[]

[Postprocessors]
  [center_temp]
    type = PointValue
    variable = T
    point = '3.5 2.5 0.0'
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [center_temp_tend]
    type = PointValue
    variable = T
    point = '3.5 2.5 0.0'
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [env_temp]
    type = FunctionValuePostprocessor
    function = temp_env
    execute_on = 'INITIAL TIMESTEP_BEGIN'
  []
  [reward]
    type = FunctionValuePostprocessor
    function = reward_function
    execute_on = 'INITIAL TIMESTEP_END'
    indirect_dependencies = 'center_temp_tend env_temp'
  []
  [top_flux]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
  []
  [log_prob_top_flux]
    type = LibtorchDRLLogProbabilityPostprocessor
    control_name = src_control
  []
[]

[Reporters]
  [T_reporter]
    type = AccumulateReporter
    reporters = 'center_temp_tend/value env_temp/value reward/value top_flux/value log_prob_top_flux/value'
  []
[]

[Controls]
  inactive = 'src_control_final'
  [src_control]
    type = LibtorchDRLControl
    parameters = "BCs/top_flux/value"
    responses = 'center_temp_tend env_temp'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 2
    response_scaling_factors = '0.03 0.03'
    response_shift_factors = '290 290'
    action_standard_deviations = '0.02'
    action_scaling_factors = 200

    execute_on = 'TIMESTEP_BEGIN'
  []
  [src_control_final]
    type = LibtorchNeuralNetControl

    filename = 'mynet_control.net'
    num_neurons_per_layer = '16 6'
    activation_function = 'relu'

    parameters = "BCs/top_flux/value"
    responses = 'center_temp_tend env_temp'

    # keep consistent with LibtorchDRLControlTrainer
    input_timesteps = 2
    response_scaling_factors = '0.03 0.03'
    response_shift_factors = '290 290'
    action_standard_deviations = '0.02'
    action_scaling_factors = 200

    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Executioner]
  type = Transient
  solve_type = 'NEWTON'

  petsc_options_iname = '-pc_type -pc_factor_shift_type'
  petsc_options_value = 'lu NONZERO'
  line_search = 'none'

  nl_rel_tol = 1e-7

  start_time = 0.0
  end_time = 86400
  dt = 900.0
[]

[Outputs]
  console = false
  [c]
    type = CSV
    execute_on = FINAL
  []
[]

(modules/stochastic_tools/include/libtorch/controls/LibtorchDRLControl.h)

// This file is part of the MOOSE framework
// https://mooseframework.inl.gov
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#ifdef LIBTORCH_ENABLED

#pragma once

#include "LibtorchArtificialNeuralNet.h"
#include "LibtorchNeuralNetControl.h"

/**
 * A time-dependent, neural-network-based controller which is
 * associated with a Proximal Policy Optimization. We use this neural net for the training of a
 * controller. The additional functionality in this controller is the addition of the variability
 * (using an assumed Gaussian distribution) to avoid overfitting. This control is
 * supposed to be used in conjunction with LibtorchDRLControlTrainer.
 */
class LibtorchDRLControl : public LibtorchNeuralNetControl
{
public:
  static InputParameters validParams();

  /// Construct using input parameters
  LibtorchDRLControl(const InputParameters & parameters);

  /// We compute the actions in this function together with the corresponding logarithmic probabilities.
  virtual void execute() override;

  /**
   * Get the logarithmic probability of (signal_index)-th signal of the control neural net
   * @param signal_index The index of the signal
   * @return The logarithmic probability of the (signal_index)-th signal
   */
  Real getSignalLogProbability(const unsigned int signal_index) const;

protected:
  /**
   * Function which computes the logarithmic probability of given actions.
   * @param action The tensor containing the perturbed control signals (also known as the action of
   * the controller)
   * @param output_tensor The expected value of the signals predicted by the neural net
   * @return The logarithmic probability of the action with respect to the neural net prediction
   */
  torch::Tensor computeLogProbability(const torch::Tensor & action,
                                      const torch::Tensor & output_tensor);

  /// The log probability of control signals from the last evaluation of the controller
  std::vector<Real> _current_control_signal_log_probabilities;

  /// Standard deviation for the actions, supplied by the user
  const std::vector<Real> _action_std;

  /// Standard deviations converted to a 2D diagonal tensor that can be used by Libtorch routines.
  torch::Tensor _std;
};

#endif

Overview
Input Parameters
Input Files
Child Objects