LibtorchRLPPOLoss Class Reference

PPO clipped surrogate loss on top of the reusable RL buffer/value-estimation core. More...

#include <LibtorchRLPPOLoss.h>

Public Member Functions
	LibtorchRLPPOLoss (Real clip_parameter, Real entropy_coeff)
	Build the PPO loss helper.
LibtorchRLPPOLossOutput	compute (Moose::LibtorchActorNeuralNet &policy_network, Moose::LibtorchArtificialNeuralNet &value_network, const LibtorchRLMiniBatch &batch) const
	Compute actor, critic, and entropy terms for one mini-batch.

Static Private Member Functions
static torch::Tensor	reduceActionDimension (const torch::Tensor &tensor)
	Collapse multi-action log-probabilities or entropies into one column tensor.

Private Attributes
const Real	_clip_parameter
	PPO clipping width used in the actor surrogate objective.
const Real	_entropy_coeff
	Weight applied to the entropy bonus inside the actor loss.

Detailed Description

PPO clipped surrogate loss on top of the reusable RL buffer/value-estimation core.

Definition at line 34 of file LibtorchRLPPOLoss.h.

Constructor & Destructor Documentation

LibtorchRLPPOLoss()

LibtorchRLPPOLoss::LibtorchRLPPOLoss	(	Real	clip_parameter,
		Real	entropy_coeff
	)

Build the PPO loss helper.

Parameters

clip_parameter	PPO clipping width.
entropy_coeff	Weight applied to the entropy bonus.

Definition at line 14 of file LibtorchRLPPOLoss.C.

  : _clip_parameter(clip_parameter), _entropy_coeff(entropy_coeff)
{
}

Member Function Documentation

compute()

LibtorchRLPPOLossOutput LibtorchRLPPOLoss::compute	(	Moose::LibtorchActorNeuralNet &	policy_network,
		Moose::LibtorchArtificialNeuralNet &	value_network,
		const LibtorchRLMiniBatch &	batch
	)		const

Compute actor, critic, and entropy terms for one mini-batch.

Parameters

policy_network	Actor network used for the policy term.
value_network	Critic network used for the value term.
batch	Mini-batch pulled from the on-policy trajectory buffer.

Definition at line 20 of file LibtorchRLPPOLoss.C.

{
  auto observations = batch.observations;
  policy_network.evaluate(observations, false);
 
  const auto current_log_probability =
      reduceActionDimension(policy_network.logProbability(batch.actions));
  const auto previous_log_probability = reduceActionDimension(batch.old_log_probabilities);
  const auto entropy = reduceActionDimension(policy_network.entropy());
 
  const auto ratio = (current_log_probability - previous_log_probability).exp();
  const auto surr1 = ratio * batch.advantages;
  const auto surr2 =
      torch::clamp(ratio, 1.0 - _clip_parameter, 1.0 + _clip_parameter) * batch.advantages;
 
  LibtorchRLPPOLossOutput output;
  output.actor_loss = -(torch::min(surr1, surr2) + _entropy_coeff * entropy).mean();
  output.critic_loss =
      torch::mse_loss(value_network.forward(batch.observations), batch.value_targets);
  output.entropy = entropy.mean();
  return output;
}

Referenced by LibtorchDRLControlTrainer::trainController().

reduceActionDimension()

torch::Tensor LibtorchRLPPOLoss::reduceActionDimension ( const torch::Tensor &  tensor )

staticprivate

Collapse multi-action log-probabilities or entropies into one column tensor.

Parameters

tensor

Action-wise tensor to reduce.

Definition at line 46 of file LibtorchRLPPOLoss.C.

{
  if (!tensor.defined())
    return tensor;
 
  if (tensor.dim() == 1)
    return tensor.unsqueeze(1);
 
  if (tensor.size(-1) == 1)
    return tensor;
 
  return tensor.sum(-1, true);
}

Referenced by compute().

Member Data Documentation

_clip_parameter

const Real LibtorchRLPPOLoss::_clip_parameter

private

PPO clipping width used in the actor surrogate objective.

Definition at line 62 of file LibtorchRLPPOLoss.h.

Referenced by compute().

_entropy_coeff

const Real LibtorchRLPPOLoss::_entropy_coeff

private

Weight applied to the entropy bonus inside the actor loss.

Definition at line 64 of file LibtorchRLPPOLoss.h.

Referenced by compute().

---

The documentation for this class was generated from the following files:

• stochastic_tools/include/libtorch/utils/LibtorchRLPPOLoss.h
• stochastic_tools/src/libtorch/utils/LibtorchRLPPOLoss.C