LibtorchRLValueEstimator Class Reference

Computes generalized-advantage estimates and value targets for an on-policy trajectory buffer, following Schulman et al., "High-Dimensional Continuous Control Using Generalized Advantage Estimation. More...

#include <LibtorchRLValueEstimator.h>

Classes
struct	Targets
	Generalized-advantage estimates and critic targets for one trajectory. More...

Public Member Functions
	LibtorchRLValueEstimator (Real discount_factor, Real lambda_factor)
	Build the GAE helper.
void	computeValueTargets (LibtorchRLTrajectoryBuffer &buffer, Moose::LibtorchArtificialNeuralNet &value_network) const
	Fill every trajectory in the buffer with value targets and advantages.
Targets	estimate (const LibtorchRLTrajectoryBuffer::Trajectory &trajectory, Moose::LibtorchArtificialNeuralNet &value_network) const
	Compute value targets and advantages for one trajectory.

Private Member Functions
std::vector< Real >	evaluate (const std::vector< std::vector< Real > > &observations, Moose::LibtorchArtificialNeuralNet &value_network) const
	Evaluate the critic on a batch of observations.

Private Attributes
const Real	_discount_factor
	Reward discount factor used in the temporal-difference recursion.
const Real	_lambda_factor
	GAE lambda factor used in the reverse-time advantage recursion.

Detailed Description

Computes generalized-advantage estimates and value targets for an on-policy trajectory buffer, following Schulman et al., "High-Dimensional Continuous Control Using Generalized Advantage Estimation.

"

Definition at line 24 of file LibtorchRLValueEstimator.h.

Constructor & Destructor Documentation

LibtorchRLValueEstimator()

LibtorchRLValueEstimator::LibtorchRLValueEstimator	(	Real	discount_factor,
		Real	lambda_factor
	)

Build the GAE helper.

Parameters

discount_factor	Reward discount factor.
lambda_factor	GAE lambda factor.

Definition at line 41 of file LibtorchRLValueEstimator.C.

  : _discount_factor(discount_factor), _lambda_factor(lambda_factor)
{
}

Member Function Documentation

computeValueTargets()

void LibtorchRLValueEstimator::computeValueTargets	(	LibtorchRLTrajectoryBuffer &	buffer,
		Moose::LibtorchArtificialNeuralNet &	value_network
	)		const

Fill every trajectory in the buffer with value targets and advantages.

Parameters

buffer	On-policy trajectory buffer to update.
value_network	Critic used for target estimation.

Definition at line 48 of file LibtorchRLValueEstimator.C.

{
  for (auto & trajectory : buffer.trajectories())
  {
    const auto targets = estimate(trajectory, value_network);
    trajectory.advantages = targets.advantages;
    trajectory.value_targets = targets.value_targets;
  }
}

Referenced by LibtorchDRLControlTrainer::execute().

estimate()

LibtorchRLValueEstimator::Targets LibtorchRLValueEstimator::estimate	(	const LibtorchRLTrajectoryBuffer::Trajectory &	trajectory,
		Moose::LibtorchArtificialNeuralNet &	value_network
	)		const

Compute value targets and advantages for one trajectory.

Parameters

trajectory	Trajectory to evaluate.
value_network	Critic used for target estimation.

Definition at line 60 of file LibtorchRLValueEstimator.C.

{
  Targets targets;
 
  const auto values = evaluate(trajectory.observations, value_network);
  const auto next_values = evaluate(trajectory.next_observations, value_network);
 
  const auto num_steps = trajectory.rewards.size();
  targets.advantages.resize(num_steps, 0.0);
  targets.value_targets.resize(num_steps, 0.0);
 
  Real gae = 0.0;
  for (const auto reverse_step : make_range(num_steps))
  {
    const auto step = num_steps - reverse_step - 1;
    const Real continuation =
        trajectory.terminals.empty() || !trajectory.terminals[step] ? 1.0 : 0.0;
    const auto delta = trajectory.rewards[step] +
                       _discount_factor * continuation * next_values[step] - values[step];
    gae = delta + _discount_factor * _lambda_factor * continuation * gae;
    targets.advantages[step] = gae;
    targets.value_targets[step] = gae + values[step];
  }
 
  return targets;
}

Referenced by computeValueTargets().

evaluate()

std::vector< Real > LibtorchRLValueEstimator::evaluate ( const std::vector< std::vector< Real > > &  observations, Moose::LibtorchArtificialNeuralNet &  value_network  ) const

private

Evaluate the critic on a batch of observations.

Parameters

observations	Observation matrix to feed through the critic.
value_network	Critic used for the evaluation.

Definition at line 89 of file LibtorchRLValueEstimator.C.

{
  if (observations.empty())
    return {};
 
  torch::NoGradGuard no_grad;
 
  auto tensor = valueEstimatorMatrixToTensor(observations);
  auto value_tensor = value_network.forward(tensor);
  auto flattened_value_tensor = value_tensor.reshape({-1});
 
  std::vector<Real> values;
  LibtorchUtils::tensorToVector(flattened_value_tensor, values);
  return values;
}

Referenced by estimate().

Member Data Documentation

_discount_factor

const Real LibtorchRLValueEstimator::_discount_factor

private

Reward discount factor used in the temporal-difference recursion.

Definition at line 71 of file LibtorchRLValueEstimator.h.

Referenced by estimate().

_lambda_factor

const Real LibtorchRLValueEstimator::_lambda_factor

private

GAE lambda factor used in the reverse-time advantage recursion.

Definition at line 73 of file LibtorchRLValueEstimator.h.

Referenced by estimate().

---

The documentation for this class was generated from the following files:

• stochastic_tools/include/libtorch/utils/LibtorchRLValueEstimator.h
• stochastic_tools/src/libtorch/utils/LibtorchRLValueEstimator.C