LibtorchNeuralNetControl

Controls the value of multiple controllable input parameters using a Libtorch-based neural network.

Overview

This object controls a physical process using a neural network. The neural network can be either read from a file of transferred from a trainer. One can use the "filename" parameter to load the neural network from file. This capability supports two formats:

Reading entire neural networks using a TorchScript format in which case the user needs to specify "torch_script_format" parameter;
Reading neural network parameters from a binary (usually .net) file. In this case only parameters are read meaning that the structure of the neural network is not established in advance. For this reason, we need to specify the architecture using "num_neurons_per_layer" and "activation_function" parameters.

Input Parameters

observationsPostprocessors used as the current observation vector.
C++ Type:std::vector<PostprocessorName>
Unit:(no unit assumed)
Controllable:No
Description:Postprocessors used as the current observation vector.
parametersControllable input parameters driven by the network.
C++ Type:std::vector<std::string>
Controllable:No
Description:Controllable input parameters driven by the network.

Required Parameters

action_scaling_factorsPer-action scaling embedded in the controller outputs so saved checkpoints stay in physical units.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Per-action scaling embedded in the controller outputs so saved checkpoints stay in physical units.
activation_functionrelu Activation name for each hidden layer, or one shared value for all layers.
Default:relu
C++ Type:std::vector<std::string>
Controllable:No
Description:Activation name for each hidden layer, or one shared value for all layers.
depends_onThe Controls that this control relies upon (i.e. must execute before this one)
C++ Type:std::vector<std::string>
Controllable:No
Description:The Controls that this control relies upon (i.e. must execute before this one)
execute_onINITIAL TIMESTEP_ENDThe list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
Default:INITIAL TIMESTEP_END
C++ Type:ExecFlagEnum
Options:XFEM_MARK, FORWARD, ADJOINT, HOMOGENEOUS_FORWARD, ADJOINT_TIMESTEP_BEGIN, ADJOINT_TIMESTEP_END, NONE, INITIAL, LINEAR, LINEAR_CONVERGENCE, NONLINEAR, NONLINEAR_CONVERGENCE, POSTCHECK, TIMESTEP_END, TIMESTEP_BEGIN, MULTIAPP_FIXED_POINT_END, MULTIAPP_FIXED_POINT_BEGIN, MULTIAPP_FIXED_POINT_CONVERGENCE, MULTISYSTEM_FIXED_POINT_ITERATION_END, FINAL, CUSTOM, PRE_MULTIAPP_SETUP
Controllable:No
Description:The list of flag(s) indicating when this object should be executed. For a description of each flag, see https://mooseframework.inl.gov/source/interfaces/SetupInterface.html.
filenameFile containing the controller network.
C++ Type:std::string
Controllable:No
Description:File containing the controller network.
input_timesteps1Number of recent timesteps to stack into each network input.
Default:1
C++ Type:unsigned int
Range:1<=input_timesteps
Controllable:No
Description:Number of recent timesteps to stack into each network input.
num_neurons_per_layerHidden-layer widths used when constructing the controller.
C++ Type:std::vector<unsigned int>
Controllable:No
Description:Hidden-layer widths used when constructing the controller.
observation_scaling_factorsOptional multipliers applied after shifting the observation values.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional multipliers applied after shifting the observation values.
observation_shift_factorsOptional offsets applied to the observation values before scaling.
C++ Type:std::vector<Real>
Unit:(no unit assumed)
Controllable:No
Description:Optional offsets applied to the observation values before scaling.
torch_script_formatFalseWhether 'filename' is a TorchScript module to evaluate as-is. Otherwise, a network is built from 'num_neurons_per_layer' and 'activation_function', and only its parameters are read from the file.
Default:False
C++ Type:bool
Controllable:No
Description:Whether 'filename' is a TorchScript module to evaluate as-is. Otherwise, a network is built from 'num_neurons_per_layer' and 'activation_function', and only its parameters are read from the file.

Optional Parameters

control_tagsAdds user-defined labels for accessing object parameters via control logic.
C++ Type:std::vector<std::string>
Controllable:No
Description:Adds user-defined labels for accessing object parameters via control logic.
enableTrueSet the enabled status of the MooseObject.
Default:True
C++ Type:bool
Controllable:No
Description:Set the enabled status of the MooseObject.
implicitTrueDetermines whether this object is calculated using an implicit or explicit form
Default:True
C++ Type:bool
Controllable:No
Description:Determines whether this object is calculated using an implicit or explicit form

Advanced Parameters

Input Files

(test/tests/controls/libtorch_nn_control/read_control.i)

Child Objects

(modules/stochastic_tools/include/libtorch/controls/LibtorchDRLControl.h)

warning

The detailed documentation of this object is only available when Moose is compiled with Libtorch. For instructions on how to compile Moose with Libtorch, visit the general installation webpage or click here.

(test/tests/controls/libtorch_nn_control/read_control.i)

pi = 3.14159265359
period = 0.25
diff_coeff = 0.5
cp = 1.0

[Functions]
  [src_func]
    type = ParsedFunction
    expression = "sin(${pi}/${period}*t)"
  []
[]

[Mesh]
  [msh]
    type = GeneratedMeshGenerator
    dim = 2
    nx = 20
    xmin = -0.5
    xmax = 0.5
    ny = 20
    ymin = -0.5
    ymax = 0.5
  []
  [source_domain]
    type = ParsedSubdomainMeshGenerator
    input = msh
    combinatorial_geometry = '(x<0.2 & x>-0.2) & (y<0.2 & y>-0.2)'
    block_id = 1
  []
[]

[Variables]
  [T]
    initial_condition = 1
  []
[]

[Kernels]
  [diffusion]
    type = CoefDiffusion
    variable = T
    coef = ${diff_coeff}
  []
  [source]
    type = BodyForce
    variable = T
    function = src_func
    block = 1
  []
  [anti_source]
    type = BodyForce
    variable = T
    value = 0
    block = 1
  []
  [time_deriv]
    type = CoefTimeDerivative
    Coefficient = ${cp}
    variable = T
  []
[]

[BCs]
  [neumann_rest]
    type = NeumannBC
    variable = T
    boundary = 'left right top bottom'
    value = 0
  []
[]

[Executioner]
  type = Transient
  num_steps = 25
  dt = 0.1
  solve_type = NEWTON
  petsc_options_iname = '-pc_type'
  petsc_options_value = 'lu'
  nl_abs_tol = 1e-12
  line_search = 'none'
[]

[Postprocessors]
  [T_max]
    type = NodalExtremeValue
    variable = T
    execute_on = 'INITIAL TIMESTEP_END'
  []
  [control_value]
    type = LibtorchControlValuePostprocessor
    control_name = src_control
    execute_on = 'INITIAL TIMESTEP_END'
  []
[]

[Controls]
  [src_control]
    type = LibtorchNeuralNetControl
    parameters = "Kernels/anti_source/value"
    observations = 'T_max'
    execute_on = 'TIMESTEP_BEGIN'
  []
[]

[Reporters]
  inactive = 'nn_parameters'
  [T_reporter]
    type = AccumulateReporter
    reporters = 'T_max/value control_value/value'
    outputs = csv_out
  []
  [nn_parameters]
    type = LibtorchArtificialNeuralNetParameters
    control_name = src_control
    execute_on = FINAL
    outputs = json_out
  []
[]

[Outputs]
  [csv_out]
    type = CSV
    execute_on = FINAL
  []
  [json_out]
    type = JSON
    execute_on = FINAL
    execute_system_information_on = NONE
  []
[]

(modules/stochastic_tools/include/libtorch/controls/LibtorchDRLControl.h)

// This file is part of the MOOSE framework
// https://mooseframework.inl.gov
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#ifdef MOOSE_LIBTORCH_ENABLED

#pragma once

#include <cstdint>

#include "LibtorchActorNeuralNet.h"
#include "LibtorchNeuralNetControl.h"

/**
 * A time-dependent, neural-network-based controller which is
 * associated with a Proximal Policy Optimization. We use this neural net for the training of a
 * controller. The additional functionality in this controller is the addition of the variability
 * (using an assumed Gaussian distribution) to avoid overfitting. This control is
 * supposed to be used in conjunction with LibtorchDRLControlTrainer.
 */
class LibtorchDRLControl : public LibtorchNeuralNetControl
{
public:
  /**
   * Return the input parameters supported by this control.
   */
  static InputParameters validParams();

  /**
   * Construct using input parameters.
   */
  LibtorchDRLControl(const InputParameters & parameters);

  /**
   * Restore any restartable controller state after base setup completes.
   */
  virtual void initialSetup() override;

  /**
   * Compute the actions and their corresponding logarithmic probabilities.
   */
  virtual void execute() override;

  /**
   * Get the logarithmic probability of (signal_index)-th signal of the control neural net
   * @param signal_index The index of the signal
   * @return The logarithmic probability of the (signal_index)-th signal
   */
  Real getSignalLogProbability(const unsigned int signal_index) const;

  /**
   * Copy an actor network into this DRL controller.
   * @param input_nn Actor network to copy into the controller.
   */
  void loadControlNeuralNet(const Moose::LibtorchActorNeuralNet & input_nn);

  /**
   * Reject loading a plain artificial neural network into this actor-based controller.
   * @param input_nn Artificial neural network passed through the base-class interface.
   */
  virtual void loadControlNeuralNet(const Moose::LibtorchArtificialNeuralNet & input_nn) override;

  /**
   * Load the actor neural network from the configured checkpoint file.
   */
  virtual void loadControlNeuralNetFromFile() override;

  /**
   * Reset the owned policy-sampling generator to a known seed.
   * @param seed Seed assigned to the policy-sampling generator.
   */
  void setPolicySampleSeed(uint64_t seed);

  /**
   * Return the number of time steps between policy evaluations.
   */
  unsigned int numStepsInPeriod() const { return _num_steps_in_period; }

  /**
   * Return the number of observation time levels stacked into the actor input.
   */
  unsigned int inputTimesteps() const { return _input_timesteps; }

  /**
   * Return the number of observations read at each time level.
   */
  unsigned int numberOfObservations() const { return _observation_names.size(); }

protected:
  /**
   * Apply the current smoothed control signals, including optional offsets.
   */
  void applyControlSignals();

  /**
   * Return the offset to add to a control signal at the current time.
   * @param control_i Index of the control signal.
   */
  Real computeControlOffset(unsigned int control_i) const;

  /// The log probability of control signals from the last evaluation of the controller
  std::vector<Real> & _current_control_signal_log_probabilities;

  /// The smoothed control signal from the previous execution, saved for restart/recover.
  std::vector<Real> & _previous_control_signal;
  /// The current smoothed control signal applied to the controllable parameters.
  std::vector<Real> & _current_smoothed_signal;

  /// Constant offsets added to each control signal before it is applied.
  std::vector<Real> _control_offsets;
  /// Whether any offset data was provided by the user.
  bool _has_control_offsets;

  /// Actor network used when the controller operates as a stochastic policy.
  std::shared_ptr<Moose::LibtorchActorNeuralNet> _actor_nn;
  /// Owned libtorch CPU generator used for policy sampling.
  at::Generator _policy_generator;
  /// Restartable serialized state for the owned policy-sampling generator.
  std::vector<std::uint8_t> & _policy_generator_state;

  /// Number of time steps between policy evaluations.
  const unsigned int _num_steps_in_period;
  /// Exponential smoothing factor in (0, 1] applied to the control signal at each time step.
  const Real _control_smoothing_factor;
  /// Whether to sample actions stochastically instead of using the deterministic actor output.
  const bool _stochastic;

private:
  /**
   * Report whether the policy is evaluated at the current time step.
   */
  bool shouldEvaluatePolicy() const;

  /**
   * Restore the owned libtorch generator state from restartable storage.
   */
  void restorePolicyGeneratorState();

  /**
   * Mirror the owned libtorch generator state into restartable storage.
   */
  void savePolicyGeneratorState();
};

#endif

Overview
Input Parameters
Input Files
Child Objects