StochasticTools::GaussianProcess Class Reference

Utility class dedicated to hold structures and functions commont to Gaussian Processes. More...

#include <GaussianProcess.h>

Classes
struct	GPOptimizerOptions
	Structure containing the optimization options for hyperparameter-tuning. More...

Public Member Functions
	GaussianProcess ()
void	initialize (CovarianceFunctionBase *covariance_function, const std::vector< std::string > &params_to_tune, const std::vector< Real > &min=std::vector< Real >(), const std::vector< Real > &max=std::vector< Real >())
	Initializes the most important structures in the Gaussian Process: the covariance function and a tuning map which is used if the user requires parameter tuning. More...
void	setupCovarianceMatrix (const RealEigenMatrix &training_params, const RealEigenMatrix &training_data, const GPOptimizerOptions &opts)
	Sets up the covariance matrix given data and optimization options. More...
void	setupStoredMatrices (const RealEigenMatrix &input)
	Sets up the Cholesky decomposition and inverse action of the covariance matrix. More...
void	linkCovarianceFunction (CovarianceFunctionBase *covariance_function)
	Finds and links the covariance function to this object. More...
void	generateTuningMap (const std::vector< std::string > &params_to_tune, const std::vector< Real > &min=std::vector< Real >(), const std::vector< Real > &max=std::vector< Real >())
	Sets up the tuning map which is used if the user requires parameter tuning. More...
void	standardizeParameters (RealEigenMatrix &parameters, bool keep_moments=false)
	Standardizes the vector of input parameters (x values). More...
void	standardizeData (RealEigenMatrix &data, bool keep_moments=false)
	Standardizes the vector of responses (y values). More...
void	tuneHyperParamsAdam (const RealEigenMatrix &training_params, const RealEigenMatrix &training_data, const GPOptimizerOptions &opts)
Real	getLoss (RealEigenMatrix &inputs, RealEigenMatrix &outputs)
std::vector< Real >	getGradient (RealEigenMatrix &inputs) const
void	mapToVec (const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &tuning_data, const std::unordered_map< std::string, Real > &scalar_map, const std::unordered_map< std::string, std::vector< Real >> &vector_map, std::vector< Real > &vec) const
	Function used to convert the hyperparameter maps in this object to vectors. More...
void	vecToMap (const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &tuning_data, std::unordered_map< std::string, Real > &scalar_map, std::unordered_map< std::string, std::vector< Real >> &vector_map, const std::vector< Real > &vec) const
	Function used to convert the vectors back to hyperparameter maps. More...
const StochasticTools::Standardizer &	getParamStandardizer () const
	Get constant reference to the contained structures. More...
const StochasticTools::Standardizer &	getDataStandardizer () const
const RealEigenMatrix &	getK () const
const RealEigenMatrix &	getKResultsSolve () const
const Eigen::LLT< RealEigenMatrix > &	getKCholeskyDecomp () const
const CovarianceFunctionBase &	getCovarFunction () const
const CovarianceFunctionBase *	getCovarFunctionPtr () const
const std::string &	getCovarType () const
const std::string &	getCovarName () const
const std::vector< UserObjectName > &	getDependentCovarNames () const
const std::map< UserObjectName, std::string > &	getDependentCovarTypes () const
const unsigned int &	getCovarNumOutputs () const
const unsigned int &	getNumTunableParams () const
const std::unordered_map< std::string, Real > &	getHyperParamMap () const
const std::unordered_map< std::string, std::vector< Real > > &	getHyperParamVectorMap () const
const std::vector< Real > &	getLengthScales () const
StochasticTools::Standardizer &	paramStandardizer ()
	Get non-constant reference to the contained structures (if they need to be modified from the utside) More...
StochasticTools::Standardizer &	dataStandardizer ()
RealEigenMatrix &	K ()
RealEigenMatrix &	KResultsSolve ()
Eigen::LLT< RealEigenMatrix > &	KCholeskyDecomp ()
CovarianceFunctionBase *	covarFunctionPtr ()
CovarianceFunctionBase &	covarFunction ()
std::string &	covarType ()
std::string &	covarName ()
std::map< UserObjectName, std::string > &	dependentCovarTypes ()
std::vector< UserObjectName > &	dependentCovarNames ()
unsigned int &	covarNumOutputs ()
std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real > > &	tuningData ()
std::unordered_map< std::string, Real > &	hyperparamMap ()
std::unordered_map< std::string, std::vector< Real > > &	hyperparamVectorMap ()
std::vector< Real > &	lengthScales ()

Protected Attributes
CovarianceFunctionBase *	_covariance_function = nullptr
	Covariance function object. More...
std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real > >	_tuning_data
	Contains tuning inforation. Index of hyperparam, size, and min/max bounds. More...
unsigned int	_num_tunable
	Number of tunable hyperparameters. More...
std::string	_covar_type
	Type of covariance function used for this GP. More...
std::string	_covar_name
	The name of the covariance function used in this GP. More...
std::vector< UserObjectName >	_dependent_covar_names
	The names of the covariance functions the used covariance function depends on. More...
std::map< UserObjectName, std::string >	_dependent_covar_types
	The types of the covariance functions the used covariance function depends on. More...
unsigned int	_num_outputs
	The number of outputs of the GP. More...
std::unordered_map< std::string, Real >	_hyperparam_map
	Scalar hyperparameters. Stored for use in surrogate. More...
std::unordered_map< std::string, std::vector< Real > >	_hyperparam_vec_map
	Vector hyperparameters. Stored for use in surrogate. More...
StochasticTools::Standardizer	_param_standardizer
	Standardizer for use with params (x) More...
StochasticTools::Standardizer	_data_standardizer
	Standardizer for use with data (y) More...
RealEigenMatrix	_K
	An _n_sample by _n_sample covariance matrix constructed from the selected kernel function. More...
RealEigenMatrix	_K_results_solve
	A solve of Ax=b via Cholesky. More...
Eigen::LLT< RealEigenMatrix >	_K_cho_decomp
	Cholesky decomposition Eigen object. More...
const RealEigenMatrix *	_training_params
	Paramaters (x) used for training, along with statistics. More...
const RealEigenMatrix *	_training_data
	Data (y) used for training. More...
unsigned int	_batch_size
	The batch size for Adam optimization. More...
std::vector< Real >	_length_scales
	To return the GP length scales for active learning. More...

Detailed Description

Utility class dedicated to hold structures and functions commont to Gaussian Processes.

It can be used to standardize parameters, manipulate covariance data and compute additional stored matrices.

Definition at line 25 of file GaussianProcess.h.

Constructor & Destructor Documentation

GaussianProcess()

StochasticTools::GaussianProcess::GaussianProcess

(

)

Definition at line 46 of file GaussianProcess.C.

{}

Member Function Documentation

covarFunction()

CovarianceFunctionBase& StochasticTools::GaussianProcess::covarFunction ( )

inline

Definition at line 210 of file GaussianProcess.h.

{ return *_covariance_function; }

covarFunctionPtr()

CovarianceFunctionBase* StochasticTools::GaussianProcess::covarFunctionPtr ( )

inline

Definition at line 209 of file GaussianProcess.h.

{ return _covariance_function; }

covarName()

std::string& StochasticTools::GaussianProcess::covarName ( )

inline

Definition at line 212 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _covar_name; }

covarNumOutputs()

unsigned int& StochasticTools::GaussianProcess::covarNumOutputs ( )

inline

Definition at line 215 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _num_outputs; }

covarType()

std::string& StochasticTools::GaussianProcess::covarType ( )

inline

Definition at line 211 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _covar_type; }

dataStandardizer()

StochasticTools::Standardizer& StochasticTools::GaussianProcess::dataStandardizer ( )

inline

Definition at line 205 of file GaussianProcess.h.

Referenced by dataLoad(), dataStore(), GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{ return _data_standardizer; }

dependentCovarNames()

std::vector<UserObjectName>& StochasticTools::GaussianProcess::dependentCovarNames ( )

inline

Definition at line 214 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _dependent_covar_names; }

dependentCovarTypes()

std::map<UserObjectName, std::string>& StochasticTools::GaussianProcess::dependentCovarTypes ( )

inline

Definition at line 213 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _dependent_covar_types; }

generateTuningMap()

void StochasticTools::GaussianProcess::generateTuningMap	(	const std::vector< std::string > &	params_to_tune,
		const std::vector< Real > &	min = std::vector<Real>(),
		const std::vector< Real > &	max = std::vector<Real>()
	)

Sets up the tuning map which is used if the user requires parameter tuning.

Parameters

params_to_tune	List of parameters which need to be tuned.
min	List of lower bounds for the parameter tuning.
max	List of upper bounds for parameter tuning.

Definition at line 102 of file GaussianProcess.C.

Referenced by initialize().

{
  _num_tunable = 0;

  const bool upper_bounds_specified = min_vector.size();
  const bool lower_bounds_specified = max_vector.size();

  for (const auto param_i : index_range(params_to_tune))
  {
    const auto & hp = params_to_tune[param_i];
    if (_covariance_function->isTunable(hp))
    {
      unsigned int size;
      Real min;
      Real max;
      // Get size and default min/max
      const bool found = _covariance_function->getTuningData(hp, size, min, max);

      if (!found)
        ::mooseError("The covariance parameter ", hp, " could not be found!");

      // Check for overridden min/max
      min = lower_bounds_specified ? min_vector[param_i] : min;
      max = upper_bounds_specified ? max_vector[param_i] : max;
      // Save data in tuple
      _tuning_data[hp] = std::make_tuple(_num_tunable, size, min, max);
      _num_tunable += size;
    }
  }
}

getCovarFunction()

const CovarianceFunctionBase& StochasticTools::GaussianProcess::getCovarFunction ( ) const

inline

Definition at line 177 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate(), and GaussianProcessTrainer::GaussianProcessTrainer().

{ return *_covariance_function; }

getCovarFunctionPtr()

const CovarianceFunctionBase* StochasticTools::GaussianProcess::getCovarFunctionPtr ( ) const

inline

Definition at line 178 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::setupCovariance().

{ return _covariance_function; }

getCovarName()

const std::string& StochasticTools::GaussianProcess::getCovarName ( ) const

inline

Definition at line 180 of file GaussianProcess.h.

{ return _covar_name; }

getCovarNumOutputs()

const unsigned int& StochasticTools::GaussianProcess::getCovarNumOutputs ( ) const

inline

Definition at line 189 of file GaussianProcess.h.

{ return _num_outputs; }

getCovarType()

const std::string& StochasticTools::GaussianProcess::getCovarType ( ) const

inline

Definition at line 179 of file GaussianProcess.h.

{ return _covar_type; }

getDataStandardizer()

const StochasticTools::Standardizer& StochasticTools::GaussianProcess::getDataStandardizer ( ) const

inline

Definition at line 173 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate(), and ActiveLearningGaussianProcess::getTrainingStandardizer().

{ return _data_standardizer; }

getDependentCovarNames()

const std::vector<UserObjectName>& StochasticTools::GaussianProcess::getDependentCovarNames ( ) const

inline

Definition at line 181 of file GaussianProcess.h.

  {
    return _dependent_covar_names;
  }

getDependentCovarTypes()

const std::map<UserObjectName, std::string>& StochasticTools::GaussianProcess::getDependentCovarTypes ( ) const

inline

Definition at line 185 of file GaussianProcess.h.

  {
    return _dependent_covar_types;
  }

getGradient()

std::vector< Real > StochasticTools::GaussianProcess::getGradient

(

RealEigenMatrix &

inputs

)

const

Definition at line 262 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  RealEigenMatrix dKdhp(_batch_size, _batch_size);
  RealEigenMatrix alpha = _K_results_solve * _K_results_solve.transpose();
  std::vector<Real> grad_vec;
  grad_vec.resize(_num_tunable);
  for (auto iter = _tuning_data.begin(); iter != _tuning_data.end(); ++iter)
  {
    std::string hyper_param_name = iter->first;
    const auto first_index = std::get<0>(iter->second);
    const auto num_entries = std::get<1>(iter->second);
    for (unsigned int ii = 0; ii < num_entries; ++ii)
    {
      const auto global_index = first_index + ii;
      _covariance_function->computedKdhyper(dKdhp, inputs, hyper_param_name, ii);
      RealEigenMatrix tmp = alpha * dKdhp - _K_cho_decomp.solve(dKdhp);
      grad_vec[global_index] = -tmp.trace() / 2.0;
    }
  }
  return grad_vec;
}

getHyperParamMap()

const std::unordered_map<std::string, Real>& StochasticTools::GaussianProcess::getHyperParamMap ( ) const

inline

Definition at line 191 of file GaussianProcess.h.

Referenced by GaussianProcessData::initialize().

{ return _hyperparam_map; }

getHyperParamVectorMap()

const std::unordered_map<std::string, std::vector<Real> >& StochasticTools::GaussianProcess::getHyperParamVectorMap ( ) const

inline

Definition at line 192 of file GaussianProcess.h.

Referenced by GaussianProcessData::initialize().

  {
    return _hyperparam_vec_map;
  }

getK()

const RealEigenMatrix& StochasticTools::GaussianProcess::getK ( ) const

inline

Definition at line 174 of file GaussianProcess.h.

{ return _K; }

getKCholeskyDecomp()

const Eigen::LLT<RealEigenMatrix>& StochasticTools::GaussianProcess::getKCholeskyDecomp ( ) const

inline

Definition at line 176 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate().

{ return _K_cho_decomp; }

getKResultsSolve()

const RealEigenMatrix& StochasticTools::GaussianProcess::getKResultsSolve ( ) const

inline

Definition at line 175 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate().

{ return _K_results_solve; }

getLengthScales()

const std::vector<Real>& StochasticTools::GaussianProcess::getLengthScales ( ) const

inline

Definition at line 196 of file GaussianProcess.h.

Referenced by ActiveLearningGaussianProcess::getLengthScales().

{ return _length_scales; }

getLoss()

Real StochasticTools::GaussianProcess::getLoss	(	RealEigenMatrix &	inputs,
		RealEigenMatrix &	outputs
	)

Definition at line 245 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  _covariance_function->computeCovarianceMatrix(_K, inputs, inputs, true);

  RealEigenMatrix flattened_data = outputs.reshaped(outputs.rows() * outputs.cols(), 1);

  setupStoredMatrices(flattened_data);

  Real log_likelihood = 0;
  log_likelihood += -(flattened_data.transpose() * _K_results_solve)(0, 0);
  log_likelihood += -std::log(_K.determinant());
  log_likelihood -= _batch_size * std::log(2 * M_PI);
  log_likelihood = -log_likelihood / 2;
  return log_likelihood;
}

getNumTunableParams()

const unsigned int& StochasticTools::GaussianProcess::getNumTunableParams ( ) const

inline

Definition at line 190 of file GaussianProcess.h.

{ return _num_tunable; }

getParamStandardizer()

const StochasticTools::Standardizer& StochasticTools::GaussianProcess::getParamStandardizer ( ) const

inline

Get constant reference to the contained structures.

Definition at line 172 of file GaussianProcess.h.

Referenced by GaussianProcessSurrogate::evaluate().

{ return _param_standardizer; }

hyperparamMap()

std::unordered_map<std::string, Real>& StochasticTools::GaussianProcess::hyperparamMap ( )

inline

Definition at line 220 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _hyperparam_map; }

hyperparamVectorMap()

std::unordered_map<std::string, std::vector<Real> >& StochasticTools::GaussianProcess::hyperparamVectorMap ( )

inline

Definition at line 221 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

  {
    return _hyperparam_vec_map;
  }

initialize()

void StochasticTools::GaussianProcess::initialize	(	CovarianceFunctionBase *	covariance_function,
		const std::vector< std::string > &	params_to_tune,
		const std::vector< Real > &	min = std::vector<Real>(),
		const std::vector< Real > &	max = std::vector<Real>()
	)

Initializes the most important structures in the Gaussian Process: the covariance function and a tuning map which is used if the user requires parameter tuning.

Parameters

covariance_function	Pointer to the covariance function that needs to be used for the Gaussian Process.
params_to_tune	List of parameters which need to be tuned.
min	List of lower bounds for the parameter tuning.
max	List of upper bounds for parameter tuning.

Definition at line 49 of file GaussianProcess.C.

Referenced by ActiveLearningGaussianProcess::ActiveLearningGaussianProcess(), and GaussianProcessTrainer::GaussianProcessTrainer().

{
  linkCovarianceFunction(covariance_function);
  generateTuningMap(params_to_tune, min, max);
}

K()

RealEigenMatrix& StochasticTools::GaussianProcess::K ( )

inline

Definition at line 206 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _K; }

KCholeskyDecomp()

Eigen::LLT<RealEigenMatrix>& StochasticTools::GaussianProcess::KCholeskyDecomp ( )

inline

Definition at line 208 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _K_cho_decomp; }

KResultsSolve()

RealEigenMatrix& StochasticTools::GaussianProcess::KResultsSolve ( )

inline

Definition at line 207 of file GaussianProcess.h.

Referenced by dataLoad(), and dataStore().

{ return _K_results_solve; }

lengthScales()

std::vector<Real>& StochasticTools::GaussianProcess::lengthScales ( )

inline

Definition at line 225 of file GaussianProcess.h.

{ return _length_scales; }

linkCovarianceFunction()

void StochasticTools::GaussianProcess::linkCovarianceFunction

(

CovarianceFunctionBase *

covariance_function

)

Finds and links the covariance function to this object.

Used mainly in the covariance data action.

Parameters

covariance_function

Pointer to the covariance function that needs to be used for the Gaussian Process.

Definition at line 59 of file GaussianProcess.C.

Referenced by initialize(), and GaussianProcessSurrogate::setupCovariance().

{
  _covariance_function = covariance_function;
  _covar_type = _covariance_function->type();
  _covar_name = _covariance_function->name();
  _covariance_function->dependentCovarianceTypes(_dependent_covar_types);
  _dependent_covar_names = _covariance_function->dependentCovarianceNames();
  _num_outputs = _covariance_function->numOutputs();
}

mapToVec()

void StochasticTools::GaussianProcess::mapToVec	(	const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &	tuning_data,
		const std::unordered_map< std::string, Real > &	scalar_map,
		const std::unordered_map< std::string, std::vector< Real >> &	vector_map,
		std::vector< Real > &	vec
	)		const

Function used to convert the hyperparameter maps in this object to vectors.

Definition at line 285 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  for (auto iter : tuning_data)
  {
    const std::string & param_name = iter.first;
    const auto scalar_it = scalar_map.find(param_name);
    if (scalar_it != scalar_map.end())
      vec[std::get<0>(iter.second)] = scalar_it->second;
    else
    {
      const auto vector_it = vector_map.find(param_name);
      if (vector_it != vector_map.end())
        for (unsigned int ii = 0; ii < std::get<1>(iter.second); ++ii)
          vec[std::get<0>(iter.second) + ii] = (vector_it->second)[ii];
    }
  }
}

paramStandardizer()

StochasticTools::Standardizer& StochasticTools::GaussianProcess::paramStandardizer ( )

inline

Get non-constant reference to the contained structures (if they need to be modified from the utside)

Definition at line 204 of file GaussianProcess.h.

Referenced by dataLoad(), dataStore(), GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{ return _param_standardizer; }

setupCovarianceMatrix()

void StochasticTools::GaussianProcess::setupCovarianceMatrix	(	const RealEigenMatrix &	training_params,
		const RealEigenMatrix &	training_data,
		const GPOptimizerOptions &	opts
	)

Sets up the covariance matrix given data and optimization options.

Parameters

training_params	The training parameter values (x values) for the covariance matrix.
training_data	The training data (y values) for the inversion of the covariance matrix.
opts	The optimizer options.

Definition at line 70 of file GaussianProcess.C.

Referenced by GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{
  const bool batch_decision = opts.batch_size > 0 && (opts.batch_size <= training_params.rows());
  _batch_size = batch_decision ? opts.batch_size : training_params.rows();
  _K.resize(_num_outputs * _batch_size, _num_outputs * _batch_size);

  if (_tuning_data.size())
    tuneHyperParamsAdam(training_params, training_data, opts);

  _K.resize(training_params.rows() * training_data.cols(),
            training_params.rows() * training_data.cols());
  _covariance_function->computeCovarianceMatrix(_K, training_params, training_params, true);

  RealEigenMatrix flattened_data =
      training_data.reshaped(training_params.rows() * training_data.cols(), 1);

  // Compute the Cholesky decomposition and inverse action of the covariance matrix
  setupStoredMatrices(flattened_data);

  _covariance_function->buildHyperParamMap(_hyperparam_map, _hyperparam_vec_map);
}

setupStoredMatrices()

void StochasticTools::GaussianProcess::setupStoredMatrices

(

const RealEigenMatrix &

input

)

Sets up the Cholesky decomposition and inverse action of the covariance matrix.

Parameters

input

The vector/matrix which right multiples the inverse of the covariance matrix.

Definition at line 95 of file GaussianProcess.C.

Referenced by getLoss(), and setupCovarianceMatrix().

{
  _K_cho_decomp = _K.llt();
  _K_results_solve = _K_cho_decomp.solve(input);
}

standardizeData()

void StochasticTools::GaussianProcess::standardizeData	(	RealEigenMatrix &	data,
		bool	keep_moments = false
	)

Standardizes the vector of responses (y values).

Parameters

data	The vector/matrix of input data.
keep_moments	If previously computed or new moments are to be used.

Definition at line 144 of file GaussianProcess.C.

Referenced by GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{
  if (!keep_moments)
    _data_standardizer.computeSet(data);
  _data_standardizer.getStandardized(data);
}

standardizeParameters()

void StochasticTools::GaussianProcess::standardizeParameters	(	RealEigenMatrix &	parameters,
		bool	keep_moments = false
	)

Standardizes the vector of input parameters (x values).

Parameters

parameters	The vector/matrix of input data.
keep_moments	If previously computed or new moments are to be used.

Definition at line 136 of file GaussianProcess.C.

Referenced by GaussianProcessTrainer::postTrain(), and ActiveLearningGaussianProcess::reTrain().

{
  if (!keep_moments)
    _param_standardizer.computeSet(data);
  _param_standardizer.getStandardized(data);
}

tuneHyperParamsAdam()

void StochasticTools::GaussianProcess::tuneHyperParamsAdam	(	const RealEigenMatrix &	training_params,
		const RealEigenMatrix &	training_data,
		const GPOptimizerOptions &	opts
	)

Definition at line 152 of file GaussianProcess.C.

Referenced by setupCovarianceMatrix().

{
  std::vector<Real> theta(_num_tunable, 0.0);
  _covariance_function->buildHyperParamMap(_hyperparam_map, _hyperparam_vec_map);

  mapToVec(_tuning_data, _hyperparam_map, _hyperparam_vec_map, theta);

  // Internal params for Adam; set to the recommended values in the paper
  Real b1 = opts.b1;
  Real b2 = opts.b2;
  Real eps = opts.eps;
  static constexpr Real lambda = 1e-4;

  std::vector<Real> m0(_num_tunable, 0.0);
  std::vector<Real> v0(_num_tunable, 0.0);

  Real new_val;
  Real m_hat;
  Real v_hat;
  Real store_loss = 0.0;
  std::vector<Real> grad1;

  // Initialize randomizer
  std::vector<unsigned int> v_sequence(training_params.rows());
  std::iota(std::begin(v_sequence), std::end(v_sequence), 0);
  RealEigenMatrix inputs(_batch_size, training_params.cols());
  RealEigenMatrix outputs(_batch_size, training_data.cols());
  if (opts.show_every_nth_iteration)
    Moose::out << "OPTIMIZING GP HYPER-PARAMETERS USING Adam" << std::endl;
  for (unsigned int ss = 0; ss < opts.num_iter; ++ss)
  {
    // Shuffle data
    MooseRandom generator;
    generator.seed(0, 1980);
    generator.saveState();
    MooseUtils::shuffle<unsigned int>(v_sequence, generator, 0);
    for (unsigned int ii = 0; ii < _batch_size; ++ii)
    {
      for (unsigned int jj = 0; jj < training_params.cols(); ++jj)
        inputs(ii, jj) = training_params(v_sequence[ii], jj);

      for (unsigned int jj = 0; jj < training_data.cols(); ++jj)
        outputs(ii, jj) = training_data(v_sequence[ii], jj);
    }

    store_loss = getLoss(inputs, outputs);
    if (opts.show_every_nth_iteration && ((ss + 1) % opts.show_every_nth_iteration == 0))
      Moose::out << "Iteration: " << ss + 1 << " LOSS: " << store_loss << std::endl;
    grad1 = getGradient(inputs);
    for (auto iter = _tuning_data.begin(); iter != _tuning_data.end(); ++iter)
    {
      const auto first_index = std::get<0>(iter->second);
      const auto num_entries = std::get<1>(iter->second);
      for (unsigned int ii = 0; ii < num_entries; ++ii)
      {
        const auto global_index = first_index + ii;
        m0[global_index] = b1 * m0[global_index] + (1 - b1) * grad1[global_index];
        v0[global_index] =
            b2 * v0[global_index] + (1 - b2) * grad1[global_index] * grad1[global_index];
        m_hat = m0[global_index] / (1 - std::pow(b1, (ss + 1)));
        v_hat = v0[global_index] / (1 - std::pow(b2, (ss + 1)));
        new_val =
            theta[global_index] - 1.0 * (opts.learning_rate * m_hat / (std::sqrt(v_hat) + eps) +
                                         lambda * theta[global_index]);

        const auto min_value = std::get<2>(iter->second);
        const auto max_value = std::get<3>(iter->second);

        theta[global_index] = std::min(std::max(new_val, min_value), max_value);
      }
    }
    vecToMap(_tuning_data, _hyperparam_map, _hyperparam_vec_map, theta);
    _covariance_function->loadHyperParamMap(_hyperparam_map, _hyperparam_vec_map);
  }
  if (opts.show_every_nth_iteration)
  {
    Moose::out << "OPTIMIZED GP HYPER-PARAMETERS:" << std::endl;
    Moose::out << Moose::stringify(theta) << std::endl;
    Moose::out << "FINAL LOSS: " << store_loss << std::endl;
  }

  if (theta.size() > 0)
  {
    unsigned int count = 1;
    _length_scales.resize(_num_tunable - count);
    for (unsigned int i = 0; i < _num_tunable - count; ++i)
      _length_scales[i] = theta[i + 1];
  }
}

tuningData()

std::unordered_map<std::string, std::tuple<unsigned int, unsigned int, Real, Real> >& StochasticTools::GaussianProcess::tuningData ( )

inline

Definition at line 216 of file GaussianProcess.h.

  {
    return _tuning_data;
  }

vecToMap()

void StochasticTools::GaussianProcess::vecToMap	(	const std::unordered_map< std::string, std::tuple< unsigned int, unsigned int, Real, Real >> &	tuning_data,
		std::unordered_map< std::string, Real > &	scalar_map,
		std::unordered_map< std::string, std::vector< Real >> &	vector_map,
		const std::vector< Real > &	vec
	)		const

Function used to convert the vectors back to hyperparameter maps.

Definition at line 309 of file GaussianProcess.C.

Referenced by tuneHyperParamsAdam().

{
  for (auto iter : tuning_data)
  {
    const std::string & param_name = iter.first;
    if (scalar_map.find(param_name) != scalar_map.end())
      scalar_map[param_name] = vec[std::get<0>(iter.second)];
    else if (vector_map.find(param_name) != vector_map.end())
      for (unsigned int ii = 0; ii < std::get<1>(iter.second); ++ii)
        vector_map[param_name][ii] = vec[std::get<0>(iter.second) + ii];
  }
}

Member Data Documentation

_batch_size

unsigned int StochasticTools::GaussianProcess::_batch_size

protected

The batch size for Adam optimization.

Definition at line 281 of file GaussianProcess.h.

Referenced by getGradient(), getLoss(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_covar_name

std::string StochasticTools::GaussianProcess::_covar_name

protected

The name of the covariance function used in this GP.

Definition at line 242 of file GaussianProcess.h.

Referenced by covarName(), getCovarName(), and linkCovarianceFunction().

_covar_type

std::string StochasticTools::GaussianProcess::_covar_type

protected

Type of covariance function used for this GP.

Definition at line 239 of file GaussianProcess.h.

Referenced by covarType(), getCovarType(), and linkCovarianceFunction().

_covariance_function

CovarianceFunctionBase* StochasticTools::GaussianProcess::_covariance_function = nullptr

protected

Covariance function object.

Definition at line 230 of file GaussianProcess.h.

Referenced by covarFunction(), covarFunctionPtr(), generateTuningMap(), getCovarFunction(), getCovarFunctionPtr(), getGradient(), getLoss(), linkCovarianceFunction(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_data_standardizer

StochasticTools::Standardizer StochasticTools::GaussianProcess::_data_standardizer

protected

Standardizer for use with data (y)

Definition at line 263 of file GaussianProcess.h.

Referenced by dataStandardizer(), getDataStandardizer(), and standardizeData().

_dependent_covar_names

std::vector<UserObjectName> StochasticTools::GaussianProcess::_dependent_covar_names

protected

The names of the covariance functions the used covariance function depends on.

Definition at line 245 of file GaussianProcess.h.

Referenced by dependentCovarNames(), getDependentCovarNames(), and linkCovarianceFunction().

_dependent_covar_types

std::map<UserObjectName, std::string> StochasticTools::GaussianProcess::_dependent_covar_types

protected

The types of the covariance functions the used covariance function depends on.

Definition at line 248 of file GaussianProcess.h.

Referenced by dependentCovarTypes(), getDependentCovarTypes(), and linkCovarianceFunction().

_hyperparam_map

std::unordered_map<std::string, Real> StochasticTools::GaussianProcess::_hyperparam_map

protected

Scalar hyperparameters. Stored for use in surrogate.

Definition at line 254 of file GaussianProcess.h.

Referenced by getHyperParamMap(), hyperparamMap(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_hyperparam_vec_map

std::unordered_map<std::string, std::vector<Real> > StochasticTools::GaussianProcess::_hyperparam_vec_map

protected

Vector hyperparameters. Stored for use in surrogate.

Definition at line 257 of file GaussianProcess.h.

Referenced by getHyperParamVectorMap(), hyperparamVectorMap(), setupCovarianceMatrix(), and tuneHyperParamsAdam().

_K

RealEigenMatrix StochasticTools::GaussianProcess::_K

protected

An _n_sample by _n_sample covariance matrix constructed from the selected kernel function.

Definition at line 266 of file GaussianProcess.h.

Referenced by getK(), getLoss(), K(), setupCovarianceMatrix(), and setupStoredMatrices().

_K_cho_decomp

Eigen::LLT<RealEigenMatrix> StochasticTools::GaussianProcess::_K_cho_decomp

protected

Cholesky decomposition Eigen object.

Definition at line 272 of file GaussianProcess.h.

Referenced by getGradient(), getKCholeskyDecomp(), KCholeskyDecomp(), and setupStoredMatrices().

_K_results_solve

RealEigenMatrix StochasticTools::GaussianProcess::_K_results_solve

protected

A solve of Ax=b via Cholesky.

Definition at line 269 of file GaussianProcess.h.

Referenced by getGradient(), getKResultsSolve(), getLoss(), KResultsSolve(), and setupStoredMatrices().

_length_scales

std::vector<Real> StochasticTools::GaussianProcess::_length_scales

protected

To return the GP length scales for active learning.

Definition at line 284 of file GaussianProcess.h.

Referenced by getLengthScales(), lengthScales(), and tuneHyperParamsAdam().

_num_outputs

unsigned int StochasticTools::GaussianProcess::_num_outputs

protected

The number of outputs of the GP.

Definition at line 251 of file GaussianProcess.h.

Referenced by covarNumOutputs(), getCovarNumOutputs(), linkCovarianceFunction(), and setupCovarianceMatrix().

_num_tunable

unsigned int StochasticTools::GaussianProcess::_num_tunable

protected

Number of tunable hyperparameters.

Definition at line 236 of file GaussianProcess.h.

Referenced by generateTuningMap(), getGradient(), getNumTunableParams(), and tuneHyperParamsAdam().

_param_standardizer

StochasticTools::Standardizer StochasticTools::GaussianProcess::_param_standardizer

protected

Standardizer for use with params (x)

Definition at line 260 of file GaussianProcess.h.

Referenced by getParamStandardizer(), paramStandardizer(), and standardizeParameters().

_training_data

const RealEigenMatrix* StochasticTools::GaussianProcess::_training_data

protected

Data (y) used for training.

Definition at line 278 of file GaussianProcess.h.

_training_params

const RealEigenMatrix* StochasticTools::GaussianProcess::_training_params

protected

Paramaters (x) used for training, along with statistics.

Definition at line 275 of file GaussianProcess.h.

_tuning_data

std::unordered_map<std::string, std::tuple<unsigned int, unsigned int, Real, Real> > StochasticTools::GaussianProcess::_tuning_data

protected

Contains tuning inforation. Index of hyperparam, size, and min/max bounds.

Definition at line 233 of file GaussianProcess.h.

Referenced by generateTuningMap(), getGradient(), setupCovarianceMatrix(), tuneHyperParamsAdam(), and tuningData().

---

The documentation for this class was generated from the following files:

• stochastic_tools/include/utils/GaussianProcess.h
• stochastic_tools/src/utils/GaussianProcess.C