StochasticTools::GaussianProcess Class Reference

Utility class dedicated to hold structures and functions commont to Gaussian Processes. More...

#include <GaussianProcess.h>

Classes
struct	GPOptimizerOptions
	Structure containing the optimization options for hyperparameter-tuning. More...

Public Types
enum	OptimizerType { OptimizerType::Adam, OptimizerType::LegacyAdam }
using	HyperParameterMap = CovarianceFunctionBase::HyperParameterMap

Public Member Functions
	GaussianProcess ()
void	initialize (CovarianceFunctionBase *covariance_function, const std::vector< std::string > &params_to_tune, const std::vector< Real > &min=std::vector< Real >(), const std::vector< Real > &max=std::vector< Real >())
	Initializes the most important structures in the Gaussian Process: the covariance function and a tuning map which is used if the user requires parameter tuning. More...
void	setupCovarianceMatrix (const torch::Tensor &training_params, const torch::Tensor &training_data, const GPOptimizerOptions &opts)
	Sets up the covariance matrix given data and optimization options. More...
void	setupStoredMatrices (const torch::Tensor &input)
	Sets up the Cholesky decomposition and inverse action of the covariance matrix. More...
void	linkCovarianceFunction (CovarianceFunctionBase *covariance_function)
	Finds and links the covariance function to this object. More...
void	generateTuningMap (const std::vector< std::string > &params_to_tune, const std::vector< Real > &min=std::vector< Real >(), const std::vector< Real > &max=std::vector< Real >())
	Sets up the tuning map which is used if the user requires parameter tuning. More...
void	standardizeParameters (torch::Tensor &parameters, bool keep_moments=false)
	Standardizes the vector of input parameters (x values). More...
void	standardizeData (torch::Tensor &data, bool keep_moments=false)
	Standardizes the vector of responses (y values). More...
void	tuneHyperParamsAdam (const torch::Tensor &training_params, const torch::Tensor &training_data, const GPOptimizerOptions &opts)
Real	getLoss (torch::Tensor &inputs, torch::Tensor &outputs)
std::vector< Real >	getGradient (torch::Tensor &inputs) const
void	mapToVec (const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &tuning_data, const HyperParameterMap &hyperparam_map, std::vector< Real > &vec) const
	Function used to convert the hyperparameter map in this object to a flat vector. More...
void	vecToMap (const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &tuning_data, HyperParameterMap &hyperparam_map, const std::vector< Real > &vec) const
	Function used to convert the vector back to the hyperparameter map. More...
const StochasticTools::Standardizer &	getParamStandardizer () const
	Get constant reference to the contained structures. More...
const StochasticTools::Standardizer &	getDataStandardizer () const
const torch::Tensor &	getK () const
const torch::Tensor &	getKResultsSolve () const
const torch::Tensor &	getKCholeskyDecomp () const
const CovarianceFunctionBase &	getCovarFunction () const
const CovarianceFunctionBase *	getCovarFunctionPtr () const
const std::string &	getCovarType () const
const std::string &	getCovarName () const
const std::vector< UserObjectName > &	getDependentCovarNames () const
const std::map< UserObjectName, std::string > &	getDependentCovarTypes () const
const unsigned int &	getCovarNumOutputs () const
const unsigned int &	getNumTunableParams () const
const HyperParameterMap &	getHyperParamMap () const
const std::vector< Real > &	getLengthScales () const
StochasticTools::Standardizer &	paramStandardizer ()
	Get non-constant reference to the contained structures (if they need to be modified from the utside) More...
StochasticTools::Standardizer &	dataStandardizer ()
torch::Tensor &	K ()
torch::Tensor &	KResultsSolve ()
torch::Tensor &	KCholeskyDecomp ()
CovarianceFunctionBase *	covarFunctionPtr ()
CovarianceFunctionBase &	covarFunction ()
std::string &	covarType ()
std::string &	covarName ()
std::map< UserObjectName, std::string > &	dependentCovarTypes ()
std::vector< UserObjectName > &	dependentCovarNames ()
unsigned int &	covarNumOutputs ()
std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real > > &	tuningData ()
HyperParameterMap &	hyperparamMap ()
std::vector< Real > &	lengthScales ()

Protected Attributes
CovarianceFunctionBase *	_covariance_function = nullptr
	Covariance function object. More...
std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real > >	_tuning_data
	Contains tuning inforation. Index of hyperparam, size, and min/max bounds. More...
unsigned int	_num_tunable = 0
	Number of tunable hyperparameters. More...
std::string	_covar_type
	Type of covariance function used for this GP. More...
std::string	_covar_name
	The name of the covariance function used in this GP. More...
std::vector< UserObjectName >	_dependent_covar_names
	The names of the covariance functions the used covariance function depends on. More...
std::map< UserObjectName, std::string >	_dependent_covar_types
	The types of the covariance functions the used covariance function depends on. More...
unsigned int	_num_outputs = 0
	The number of outputs of the GP. More...
HyperParameterMap	_hyperparam_map
	Hyperparameters. Stored as tensors for use in surrogate reload/reporting. More...
StochasticTools::Standardizer	_param_standardizer
	Standardizer for use with params (x) More...
StochasticTools::Standardizer	_data_standardizer
	Standardizer for use with data (y) More...
torch::Tensor	_K
	An _n_sample by _n_sample covariance matrix constructed from the selected kernel function. More...
torch::Tensor	_K_results_solve
	A solve of Ax=b via Cholesky. More...
torch::Tensor	_K_cho_decomp
	Cholesky decomposition libtorch tensor object. More...
unsigned int	_batch_size = 0
	The batch size for Adam optimization. More...
std::vector< Real >	_length_scales
	To return the GP length scales for active learning. More...

Detailed Description

Utility class dedicated to hold structures and functions commont to Gaussian Processes.

It can be used to standardize parameters, manipulate covariance data and compute additional stored matrices.

Definition at line 27 of file GaussianProcess.h.

Member Typedef Documentation

HyperParameterMap

using StochasticTools::GaussianProcess::HyperParameterMap = CovarianceFunctionBase::HyperParameterMap

Definition at line 30 of file GaussianProcess.h.

Member Enumeration Documentation

OptimizerType

enum StochasticTools::GaussianProcess::OptimizerType

strong

Enumerator
Adam
LegacyAdam

Definition at line 34 of file GaussianProcess.h.

  {
    Adam,
    LegacyAdam
  };

Constructor & Destructor Documentation

GaussianProcess()

StochasticTools::GaussianProcess::GaussianProcess

(

)

Definition at line 129 of file GaussianProcess.C.

{}

Member Function Documentation

covarFunction()

CovarianceFunctionBase& StochasticTools::GaussianProcess::covarFunction ( )

inline

Definition at line 216 of file GaussianProcess.h.

{ return *_covariance_function; }

covarFunctionPtr()

CovarianceFunctionBase* StochasticTools::GaussianProcess::covarFunctionPtr ( )

inline

Definition at line 215 of file GaussianProcess.h.

{ return _covariance_function; }

covarName()

std::string& StochasticTools::GaussianProcess::covarName ( )

inline

Definition at line 218 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _covar_name; }

covarNumOutputs()

unsigned int& StochasticTools::GaussianProcess::covarNumOutputs ( )

inline

Definition at line 221 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _num_outputs; }

covarType()

std::string& StochasticTools::GaussianProcess::covarType ( )

inline

Definition at line 217 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _covar_type; }

dataStandardizer()

StochasticTools::Standardizer& StochasticTools::GaussianProcess::dataStandardizer ( )

inline

Definition at line 211 of file GaussianProcess.h.

Referenced by dataLoad(), dataStore(), GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{ return _data_standardizer; }

dependentCovarNames()

std::vector<UserObjectName>& StochasticTools::GaussianProcess::dependentCovarNames ( )

inline

Definition at line 220 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _dependent_covar_names; }

dependentCovarTypes()

std::map<UserObjectName, std::string>& StochasticTools::GaussianProcess::dependentCovarTypes ( )

inline

Definition at line 219 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _dependent_covar_types; }

generateTuningMap()

void StochasticTools::GaussianProcess::generateTuningMap	(	const std::vector< std::string > &	params_to_tune,
		const std::vector< Real > &	min = std::vector<Real>(),
		const std::vector< Real > &	max = std::vector<Real>()
	)

Sets up the tuning map which is used if the user requires parameter tuning.

Parameters

params_to_tune	List of parameters which need to be tuned.
min	List of lower bounds for the parameter tuning.
max	List of upper bounds for parameter tuning.

Definition at line 196 of file GaussianProcess.C.

Referenced by initialize().

{
  _num_tunable = 0;

  const bool upper_bounds_specified = min_vector.size();
  const bool lower_bounds_specified = max_vector.size();

  for (const auto param_i : index_range(params_to_tune))
  {
    const auto & hp = params_to_tune[param_i];
    if (_covariance_function->isTunable(hp))
    {
      unsigned int size;
      Real min;
      Real max;
      // Get size and default min/max
      const bool found = _covariance_function->getTuningData(hp, size, min, max);

      if (!found)
        ::mooseError("The covariance parameter ", hp, " could not be found!");

      // Check for overridden min/max
      min = lower_bounds_specified ? min_vector[param_i] : min;
      max = upper_bounds_specified ? max_vector[param_i] : max;
      // Save data in tuple
      _tuning_data[hp] = std::make_tuple(_num_tunable, size, min, max);
      _num_tunable += size;
    }
  }
}

getCovarFunction()

const CovarianceFunctionBase& StochasticTools::GaussianProcess::getCovarFunction ( ) const

inline

Definition at line 187 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate(), and GaussianProcessTrainer::GaussianProcessTrainer().

{ return *_covariance_function; }

getCovarFunctionPtr()

const CovarianceFunctionBase* StochasticTools::GaussianProcess::getCovarFunctionPtr ( ) const

inline

Definition at line 188 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::setupCovariance().

{ return _covariance_function; }

getCovarName()

const std::string& StochasticTools::GaussianProcess::getCovarName ( ) const

inline

Definition at line 190 of file GaussianProcess.h.

{ return _covar_name; }

getCovarNumOutputs()

const unsigned int& StochasticTools::GaussianProcess::getCovarNumOutputs ( ) const

inline

Definition at line 199 of file GaussianProcess.h.

{ return _num_outputs; }

getCovarType()

const std::string& StochasticTools::GaussianProcess::getCovarType ( ) const

inline

Definition at line 189 of file GaussianProcess.h.

{ return _covar_type; }

getDataStandardizer()

const StochasticTools::Standardizer& StochasticTools::GaussianProcess::getDataStandardizer ( ) const

inline

Definition at line 183 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate(), and ActiveLearningGaussianProcess::getTrainingStandardizer().

{ return _data_standardizer; }

getDependentCovarNames()

const std::vector<UserObjectName>& StochasticTools::GaussianProcess::getDependentCovarNames ( ) const

inline

Definition at line 191 of file GaussianProcess.h.

  {
    return _dependent_covar_names;
  }

getDependentCovarTypes()

const std::map<UserObjectName, std::string>& StochasticTools::GaussianProcess::getDependentCovarTypes ( ) const

inline

Definition at line 195 of file GaussianProcess.h.

  {
    return _dependent_covar_types;
  }

getGradient()

std::vector< Real > StochasticTools::GaussianProcess::getGradient

(

torch::Tensor &

inputs

)

const

Definition at line 379 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  torch::Tensor dKdhp = torch::empty({_num_outputs * _batch_size, _num_outputs * _batch_size},
                                     doubleOptionsLike(inputs));
  std::vector<Real> grad_vec;
  grad_vec.resize(_num_tunable);
  for (auto iter = _tuning_data.begin(); iter != _tuning_data.end(); ++iter)
  {
    std::string hyper_param_name = iter->first;
    const auto first_index = std::get<0>(iter->second);
    const auto num_entries = std::get<1>(iter->second);
    for (unsigned int ii = 0; ii < num_entries; ++ii)
    {
      const auto global_index = first_index + ii;
      _covariance_function->computedKdhyper(dKdhp, inputs, hyper_param_name, ii);
      const auto quadratic_form =
          torch::mm(torch::transpose(_K_results_solve, 0, 1), torch::mm(dKdhp, _K_results_solve))
              .item<Real>();
      const auto inverse_trace =
          torch::trace(torch::cholesky_solve(dKdhp, _K_cho_decomp)).item<Real>();
      grad_vec[global_index] = (inverse_trace - quadratic_form) / 2.0;
    }
  }
  return grad_vec;
}

getHyperParamMap()

const HyperParameterMap& StochasticTools::GaussianProcess::getHyperParamMap ( ) const

inline

Definition at line 201 of file GaussianProcess.h.

Referenced by GaussianProcessData::initialize().

{ return _hyperparam_map; }

getK()

const torch::Tensor& StochasticTools::GaussianProcess::getK ( ) const

inline

Definition at line 184 of file GaussianProcess.h.

{ return _K; }

getKCholeskyDecomp()

const torch::Tensor& StochasticTools::GaussianProcess::getKCholeskyDecomp ( ) const

inline

Definition at line 186 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate().

{ return _K_cho_decomp; }

getKResultsSolve()

const torch::Tensor& StochasticTools::GaussianProcess::getKResultsSolve ( ) const

inline

Definition at line 185 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate().

{ return _K_results_solve; }

getLengthScales()

const std::vector<Real>& StochasticTools::GaussianProcess::getLengthScales ( ) const

inline

Definition at line 202 of file GaussianProcess.h.

Referenced by ActiveLearningGaussianProcess::getLengthScales().

{ return _length_scales; }

getLoss()

Real StochasticTools::GaussianProcess::getLoss	(	torch::Tensor &	inputs,
		torch::Tensor &	outputs
	)

Definition at line 362 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  _covariance_function->computeCovarianceMatrix(_K, inputs, inputs, true);
  const auto flattened_data = flattenOutputData(outputs);

  setupStoredMatrices(flattened_data);

  Real log_likelihood = 0;
  log_likelihood +=
      -1 * torch::mm(torch::transpose(flattened_data, 0, 1), _K_results_solve).item<Real>();
  log_likelihood += -2.0 * torch::sum(torch::log(torch::diagonal(_K_cho_decomp))).item<Real>();
  log_likelihood -= flattened_data.size(0) * std::log(2 * M_PI);
  log_likelihood = -log_likelihood / 2;
  return log_likelihood;
}

getNumTunableParams()

const unsigned int& StochasticTools::GaussianProcess::getNumTunableParams ( ) const

inline

Definition at line 200 of file GaussianProcess.h.

{ return _num_tunable; }

getParamStandardizer()

const StochasticTools::Standardizer& StochasticTools::GaussianProcess::getParamStandardizer ( ) const

inline

Get constant reference to the contained structures.

Definition at line 182 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate().

{ return _param_standardizer; }

hyperparamMap()

HyperParameterMap& StochasticTools::GaussianProcess::hyperparamMap ( )

inline

Definition at line 226 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _hyperparam_map; }

initialize()

void StochasticTools::GaussianProcess::initialize	(	CovarianceFunctionBase *	covariance_function,
		const std::vector< std::string > &	params_to_tune,
		const std::vector< Real > &	min = std::vector<Real>(),
		const std::vector< Real > &	max = std::vector<Real>()
	)

Initializes the most important structures in the Gaussian Process: the covariance function and a tuning map which is used if the user requires parameter tuning.

Parameters

covariance_function	Pointer to the covariance function that needs to be used for the Gaussian Process.
params_to_tune	List of parameters which need to be tuned.
min	List of lower bounds for the parameter tuning.
max	List of upper bounds for parameter tuning.

Definition at line 132 of file GaussianProcess.C.

Referenced by ActiveLearningGaussianProcess::ActiveLearningGaussianProcess(), and GaussianProcessTrainer::GaussianProcessTrainer().

{
  linkCovarianceFunction(covariance_function);
  generateTuningMap(params_to_tune, min, max);
}

K()

torch::Tensor& StochasticTools::GaussianProcess::K ( )

inline

Definition at line 212 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _K; }

KCholeskyDecomp()

torch::Tensor& StochasticTools::GaussianProcess::KCholeskyDecomp ( )

inline

Definition at line 214 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _K_cho_decomp; }

KResultsSolve()

torch::Tensor& StochasticTools::GaussianProcess::KResultsSolve ( )

inline

Definition at line 213 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _K_results_solve; }

lengthScales()

std::vector<Real>& StochasticTools::GaussianProcess::lengthScales ( )

inline

Definition at line 227 of file GaussianProcess.h.

{ return _length_scales; }

linkCovarianceFunction()

void StochasticTools::GaussianProcess::linkCovarianceFunction

(

CovarianceFunctionBase *

covariance_function

)

Finds and links the covariance function to this object.

Used mainly in the covariance data action.

Parameters

covariance_function

Pointer to the covariance function that needs to be used for the Gaussian Process.

Definition at line 142 of file GaussianProcess.C.

Referenced by initialize(), and GaussianProcessSurrogate::setupCovariance().

{
  _covariance_function = covariance_function;
  _covar_type = _covariance_function->type();
  _covar_name = _covariance_function->name();
  _covariance_function->dependentCovarianceTypes(_dependent_covar_types);
  _dependent_covar_names = _covariance_function->dependentCovarianceNames();
  _num_outputs = _covariance_function->numOutputs();
}

mapToVec()

void StochasticTools::GaussianProcess::mapToVec	(	const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &	tuning_data,
		const HyperParameterMap &	hyperparam_map,
		std::vector< Real > &	vec
	)		const

Function used to convert the hyperparameter map in this object to a flat vector.

Definition at line 406 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  for (auto iter : tuning_data)
  {
    const std::string & param_name = iter.first;
    const auto tensor_it = hyperparam_map.find(param_name);
    if (tensor_it == hyperparam_map.end())
      mooseError("The covariance parameter ", param_name, " could not be found!");

    const auto values = exportHyperParameter(tensor_it->second);
    const auto num_entries = std::get<1>(iter.second);
    mooseAssert(values.size() == num_entries,
                "Hyperparameter size does not match tuning metadata.");
    for (unsigned int ii = 0; ii < num_entries; ++ii)
      vec[std::get<0>(iter.second) + ii] = values[ii];
  }
}

paramStandardizer()

StochasticTools::Standardizer& StochasticTools::GaussianProcess::paramStandardizer ( )

inline

Get non-constant reference to the contained structures (if they need to be modified from the utside)

Definition at line 210 of file GaussianProcess.h.

Referenced by dataLoad(), dataStore(), GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{ return _param_standardizer; }

setupCovarianceMatrix()

void StochasticTools::GaussianProcess::setupCovarianceMatrix	(	const torch::Tensor &	training_params,
		const torch::Tensor &	training_data,
		const GPOptimizerOptions &	opts
	)

Sets up the covariance matrix given data and optimization options.

Parameters

training_params	The training parameter values (x values) for the covariance matrix.
training_data	The training data (y values) for the inversion of the covariance matrix.
opts	The optimizer options.

Definition at line 153 of file GaussianProcess.C.

Referenced by GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{
  const auto options = doubleOptionsLike(training_params);
  const auto params = toOptions(training_params, options);
  const auto data = toOptions(training_data, options);

  mooseAssert(params.dim() == 2, "GaussianProcess training parameters must be rank-2.");
  mooseAssert(data.dim() == 2, "GaussianProcess training responses must be rank-2.");

  const auto num_samples = params.size(0);
  mooseAssert(data.size(0) == num_samples,
              "Training parameter and response sample counts must match.");
  mooseAssert(data.size(1) == _num_outputs,
              "Training response dimension does not match the covariance output dimension.");

  const bool batch_decision = opts.batch_size > 0 && (opts.batch_size <= num_samples);
  _batch_size = batch_decision ? opts.batch_size : num_samples;

  _hyperparam_map.clear();
  _covariance_function->buildHyperParamMap(_hyperparam_map);
  moveHyperParameters(_hyperparam_map, options);
  _covariance_function->loadHyperParamMap(_hyperparam_map);

  if (_tuning_data.size())
    tuneHyperParamsAdam(params, data, opts);

  _covariance_function->computeCovarianceMatrix(_K, params, params, true);
  const auto flattened_tensor = flattenOutputData(data);

  // Compute the Cholesky decomposition and inverse action of the covariance matrix.
  setupStoredMatrices(flattened_tensor);
}

setupStoredMatrices()

void StochasticTools::GaussianProcess::setupStoredMatrices

(

const torch::Tensor &

input

)

Sets up the Cholesky decomposition and inverse action of the covariance matrix.

Parameters

input

The vector/matrix which right multiples the inverse of the covariance matrix.

Definition at line 189 of file GaussianProcess.C.

Referenced by getLoss(), and setupCovarianceMatrix().

{
  _K_cho_decomp = torch::linalg_cholesky(_K);
  _K_results_solve = torch::cholesky_solve(input, _K_cho_decomp);
}

standardizeData()

void StochasticTools::GaussianProcess::standardizeData	(	torch::Tensor &	data,
		bool	keep_moments = false
	)

Standardizes the vector of responses (y values).

Parameters

data	The vector/matrix of input data.
keep_moments	If previously computed or new moments are to be used.

Definition at line 238 of file GaussianProcess.C.

Referenced by GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{
  if (!keep_moments)
    _data_standardizer.computeSet(data);
  _data_standardizer.getStandardized(data);
}

standardizeParameters()

void StochasticTools::GaussianProcess::standardizeParameters	(	torch::Tensor &	parameters,
		bool	keep_moments = false
	)

Standardizes the vector of input parameters (x values).

Parameters

parameters	The vector/matrix of input data.
keep_moments	If previously computed or new moments are to be used.

Definition at line 230 of file GaussianProcess.C.

Referenced by GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{
  if (!keep_moments)
    _param_standardizer.computeSet(data);
  _param_standardizer.getStandardized(data);
}

tuneHyperParamsAdam()

void StochasticTools::GaussianProcess::tuneHyperParamsAdam	(	const torch::Tensor &	training_params,
		const torch::Tensor &	training_data,
		const GPOptimizerOptions &	opts
	)

Definition at line 246 of file GaussianProcess.C.

Referenced by setupCovarianceMatrix().

{
  const auto options = doubleOptionsLike(training_params);
  std::vector<Real> theta_values(_num_tunable, 0.0);

  mapToVec(_tuning_data, _hyperparam_map, theta_values);

  auto theta = torch::from_blob(theta_values.data(),
                                {static_cast<long>(_num_tunable)},
                                torch::TensorOptions().dtype(at::kDouble))
                   .clone()
                   .to(options.device());

  auto adam_options = torch::optim::AdamOptions(opts.learning_rate);
  adam_options.betas(std::make_tuple(opts.b1, opts.b2));
  adam_options.eps(opts.eps);
  // The legacy MOOSE shrink term is decoupled and not learning-rate-scaled, so it cannot be
  // represented by Adam's coupled weight_decay option.
  adam_options.weight_decay(0.0);
  torch::optim::Adam optimizer({theta}, adam_options);

  Real store_loss = 0.0;
  std::vector<Real> grad_values;
  const bool use_legacy_update = opts.optimizer_type == OptimizerType::LegacyAdam;

  const bool use_full_batch = _batch_size == static_cast<unsigned int>(training_params.size(0));
  // Preserve the existing deterministic shuffle sequence for mini-batches, but avoid rebuilding
  // shuffled full-batch tensors when the batch already contains every training sample.
  std::vector<unsigned int> v_sequence;
  if (!use_full_batch)
  {
    v_sequence.resize(training_params.size(0));
    std::iota(std::begin(v_sequence), std::end(v_sequence), 0);
  }
  if (opts.show_every_nth_iteration)
    Moose::out << "OPTIMIZING GP HYPER-PARAMETERS USING "
               << (use_legacy_update ? "legacy-compatible Adam" : "Adam") << std::endl;
  for (unsigned int ss = 0; ss < opts.num_iter; ++ss)
  {
    torch::Tensor inputs;
    torch::Tensor outputs;
    if (use_full_batch)
    {
      inputs = training_params;
      outputs = training_data;
    }
    else
    {
      MooseRandom generator;
      generator.seed(0, 1980);
      generator.saveState();
      MooseUtils::shuffle<unsigned int>(v_sequence, generator, 0);

      std::vector<int64_t> batch_indices_vec(v_sequence.begin(), v_sequence.begin() + _batch_size);
      auto batch_indices = torch::tensor(
          batch_indices_vec, torch::TensorOptions().dtype(torch::kLong).device(options.device()));
      inputs = torch::index_select(training_params, 0, batch_indices);
      outputs = torch::index_select(training_data, 0, batch_indices);
    }

    store_loss = getLoss(inputs, outputs);
    if (opts.show_every_nth_iteration && ((ss + 1) % opts.show_every_nth_iteration == 0))
      Moose::out << "Iteration: " << ss + 1 << " LOSS: " << store_loss << std::endl;

    grad_values = getGradient(inputs);
    auto grad = torch::from_blob(grad_values.data(),
                                 {static_cast<long>(_num_tunable)},
                                 torch::TensorOptions().dtype(at::kDouble))
                    .clone()
                    .to(options.device());
    optimizer.zero_grad();
    theta.mutable_grad() = grad;
    torch::Tensor theta_before_step;
    if (use_legacy_update)
      theta_before_step = theta.detach().clone();
    optimizer.step();

    {
      torch::NoGradGuard no_grad;
      if (use_legacy_update)
        theta -= opts.lambda * theta_before_step;
      for (auto iter = _tuning_data.begin(); iter != _tuning_data.end(); ++iter)
      {
        const auto first_index = std::get<0>(iter->second);
        const auto num_entries = std::get<1>(iter->second);
        const auto min_value = std::get<2>(iter->second);
        const auto max_value = std::get<3>(iter->second);
        theta.slice(0, first_index, first_index + num_entries).clamp_(min_value, max_value);
      }
    }

    const auto theta_export = LibtorchUtils::toCPUContiguous(theta);
    const auto * theta_data = theta_export.data_ptr<Real>();
    theta_values.assign(theta_data, theta_data + theta_export.numel());
    vecToMap(_tuning_data, _hyperparam_map, theta_values);
    _covariance_function->loadHyperParamMap(_hyperparam_map);
  }
  if (opts.show_every_nth_iteration)
  {
    Moose::out << "OPTIMIZED GP HYPER-PARAMETERS:" << std::endl;
    Moose::out << Moose::stringify(theta_values) << std::endl;
    Moose::out << "FINAL LOSS: " << store_loss << std::endl;
  }

  if (theta_values.size() > 0)
  {
    unsigned int count = 1;
    _length_scales.resize(_num_tunable - count);
    for (unsigned int i = 0; i < _num_tunable - count; ++i)
      _length_scales[i] = theta_values[i + 1];
  }
}

tuningData()

std::unordered_map<std::string, std::tuple<unsigned int, unsigned int, Real, Real> >& StochasticTools::GaussianProcess::tuningData ( )

inline

Definition at line 222 of file GaussianProcess.h.

  {
    return _tuning_data;
  }

vecToMap()

void StochasticTools::GaussianProcess::vecToMap	(	const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &	tuning_data,
		HyperParameterMap &	hyperparam_map,
		const std::vector< Real > &	vec
	)		const

Function used to convert the vector back to the hyperparameter map.

Definition at line 429 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  for (auto iter : tuning_data)
  {
    const std::string & param_name = iter.first;
    const auto tensor_it = hyperparam_map.find(param_name);
    if (tensor_it == hyperparam_map.end())
      mooseError("The covariance parameter ", param_name, " could not be found!");

    const auto first_index = std::get<0>(iter.second);
    const auto num_entries = std::get<1>(iter.second);
    std::vector<Real> values(num_entries);
    for (unsigned int ii = 0; ii < num_entries; ++ii)
      values[ii] = vec[first_index + ii];

    updateHyperParameter(tensor_it->second, values, param_name);
  }
}

Member Data Documentation

_batch_size

unsigned int StochasticTools::GaussianProcess::_batch_size = 0

protected

The batch size for Adam optimization.

Definition at line 274 of file GaussianProcess.h.

Referenced by getGradient(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_covar_name

std::string StochasticTools::GaussianProcess::_covar_name

protected

The name of the covariance function used in this GP.

Definition at line 244 of file GaussianProcess.h.

Referenced by covarName(), getCovarName(), and linkCovarianceFunction().

_covar_type

std::string StochasticTools::GaussianProcess::_covar_type

protected

Type of covariance function used for this GP.

Definition at line 241 of file GaussianProcess.h.

Referenced by covarType(), getCovarType(), and linkCovarianceFunction().

_covariance_function

CovarianceFunctionBase* StochasticTools::GaussianProcess::_covariance_function = nullptr

protected

Covariance function object.

Definition at line 232 of file GaussianProcess.h.

Referenced by covarFunction(), covarFunctionPtr(), generateTuningMap(), getCovarFunction(), getCovarFunctionPtr(), getGradient(), getLoss(), linkCovarianceFunction(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_data_standardizer

StochasticTools::Standardizer StochasticTools::GaussianProcess::_data_standardizer

protected

Standardizer for use with data (y)

Definition at line 262 of file GaussianProcess.h.

Referenced by dataStandardizer(), getDataStandardizer(), and standardizeData().

_dependent_covar_names

std::vector<UserObjectName> StochasticTools::GaussianProcess::_dependent_covar_names

protected

The names of the covariance functions the used covariance function depends on.

Definition at line 247 of file GaussianProcess.h.

Referenced by dependentCovarNames(), getDependentCovarNames(), and linkCovarianceFunction().

_dependent_covar_types

std::map<UserObjectName, std::string> StochasticTools::GaussianProcess::_dependent_covar_types

protected

The types of the covariance functions the used covariance function depends on.

Definition at line 250 of file GaussianProcess.h.

Referenced by dependentCovarTypes(), getDependentCovarTypes(), and linkCovarianceFunction().

_hyperparam_map

HyperParameterMap StochasticTools::GaussianProcess::_hyperparam_map

protected

Hyperparameters. Stored as tensors for use in surrogate reload/reporting.

Definition at line 256 of file GaussianProcess.h.

Referenced by getHyperParamMap(), hyperparamMap(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_K

torch::Tensor StochasticTools::GaussianProcess::_K

protected

An _n_sample by _n_sample covariance matrix constructed from the selected kernel function.

Definition at line 265 of file GaussianProcess.h.

Referenced by getK(), getLoss(), K(), setupCovarianceMatrix(), and setupStoredMatrices().

_K_cho_decomp

torch::Tensor StochasticTools::GaussianProcess::_K_cho_decomp

protected

Cholesky decomposition libtorch tensor object.

Definition at line 271 of file GaussianProcess.h.

Referenced by getGradient(), getKCholeskyDecomp(), getLoss(), KCholeskyDecomp(), and setupStoredMatrices().

_K_results_solve

torch::Tensor StochasticTools::GaussianProcess::_K_results_solve

protected

A solve of Ax=b via Cholesky.

Definition at line 268 of file GaussianProcess.h.

Referenced by getGradient(), getKResultsSolve(), getLoss(), KResultsSolve(), and setupStoredMatrices().

_length_scales

std::vector<Real> StochasticTools::GaussianProcess::_length_scales

protected

To return the GP length scales for active learning.

Definition at line 277 of file GaussianProcess.h.

Referenced by getLengthScales(), lengthScales(), and tuneHyperParamsAdam().

_num_outputs

unsigned int StochasticTools::GaussianProcess::_num_outputs = 0

protected

The number of outputs of the GP.

Definition at line 253 of file GaussianProcess.h.

Referenced by covarNumOutputs(), getCovarNumOutputs(), getGradient(), linkCovarianceFunction(), and setupCovarianceMatrix().

_num_tunable

unsigned int StochasticTools::GaussianProcess::_num_tunable = 0

protected

Number of tunable hyperparameters.

Definition at line 238 of file GaussianProcess.h.

Referenced by generateTuningMap(), getGradient(), getNumTunableParams(), and tuneHyperParamsAdam().

_param_standardizer

StochasticTools::Standardizer StochasticTools::GaussianProcess::_param_standardizer

protected

Standardizer for use with params (x)

Definition at line 259 of file GaussianProcess.h.

Referenced by getParamStandardizer(), paramStandardizer(), and standardizeParameters().

_tuning_data

std::unordered_map<std::string, std::tuple<unsigned int, unsigned int, Real, Real> > StochasticTools::GaussianProcess::_tuning_data

protected

Contains tuning inforation. Index of hyperparam, size, and min/max bounds.

Definition at line 235 of file GaussianProcess.h.

Referenced by generateTuningMap(), getGradient(), setupCovarianceMatrix(), tuneHyperParamsAdam(), and tuningData().

---

The documentation for this class was generated from the following files:

• stochastic_tools/include/utils/GaussianProcess.h
• stochastic_tools/src/utils/GaussianProcess.C