LibtorchRLTrajectoryAssembler Class Reference

Builds fixed-horizon RL trajectories from raw per-sample reporter sequences. More...

#include <LibtorchRLTrajectoryAssembler.h>

Public Types
using	ReporterData = std::vector< const std::vector< std::vector< Real > > * >
	Raw reporter data for one rollout quantity, indexed as [component][sample][raw time].

Public Member Functions
	LibtorchRLTrajectoryAssembler (unsigned int input_timesteps, unsigned int timestep_window, bool shift_outputs, bool average_reward_over_timestep_window)
	Build the trajectory assembler.
std::vector< LibtorchRLTrajectoryBuffer::Trajectory >	assemble (const ReporterData &observations, const ReporterData &actions, const ReporterData &log_probabilities, const std::vector< std::vector< Real > > &rewards) const
	Assemble one trajectory per sample that holds at least one transition.
unsigned int	numTransitions (std::size_t raw_sequence_size) const
	Count the transitions available in a raw reporter sequence.

Private Member Functions
std::vector< Real >	downsample (const std::vector< Real > &sample, unsigned int offset, unsigned int num_entries) const
	Take every timestep_window-th entry of a raw reporter sequence.
std::vector< Real >	windowAverage (const std::vector< Real > &sample, unsigned int num_entries) const
	Average a raw reporter sequence over consecutive timestep windows after the initial entry.

Static Private Member Functions
static void	checkSampleCount (const ReporterData &data, const std::string &quantity, std::size_t num_samples)
	Check that every component of a rollout quantity holds one sequence per sample.

Private Attributes
const unsigned int	_timestep_window
	Raw reporter entries per transition.
const bool	_shift_outputs
	Whether actions and log probabilities are read one raw entry after the observation.
const bool	_average_reward_over_timestep_window
	Whether rewards are averaged over each window instead of sampled at its end.
const LibtorchObservationHistoryHelper	_observation_history
	Observation history stacking helper.

Detailed Description

Builds fixed-horizon RL trajectories from raw per-sample reporter sequences.

Raw index 0 of every sequence holds the initial value and raw index n holds the value at the end of time step n. Transition k uses the observation at raw index k * W, the action and log probability at raw index k * W (+ 1 when outputs are shifted), and the reward at raw index (k + 1) * W, or the mean over raw indices k * W + 1 through (k + 1) * W. Here W is the timestep window. Observation history lags also span W raw entries.

Definition at line 29 of file LibtorchRLTrajectoryAssembler.h.

Member Typedef Documentation

ReporterData

using LibtorchRLTrajectoryAssembler::ReporterData = std::vector<const std::vector<std::vector<Real> > *>

Raw reporter data for one rollout quantity, indexed as [component][sample][raw time].

Definition at line 33 of file LibtorchRLTrajectoryAssembler.h.

Constructor & Destructor Documentation

LibtorchRLTrajectoryAssembler()

LibtorchRLTrajectoryAssembler::LibtorchRLTrajectoryAssembler	(	unsigned int	input_timesteps,
		unsigned int	timestep_window,
		bool	shift_outputs,
		bool	average_reward_over_timestep_window
	)

Build the trajectory assembler.

Parameters

input_timesteps	Number of stacked observation time levels.
timestep_window	Raw reporter entries per transition.
shift_outputs	Whether actions and log probabilities are read one raw entry later.
average_reward_over_timestep_window	Whether rewards are averaged over each window.

Definition at line 20 of file LibtorchRLTrajectoryAssembler.C.

  : _timestep_window(timestep_window),
    _shift_outputs(shift_outputs),
    _average_reward_over_timestep_window(average_reward_over_timestep_window),
    _observation_history(input_timesteps)
{
  if (!_timestep_window)
    mooseError("The RL trajectory timestep window must be at least 1.");
}

Member Function Documentation

assemble()

std::vector< LibtorchRLTrajectoryBuffer::Trajectory > LibtorchRLTrajectoryAssembler::assemble	(	const ReporterData &	observations,
		const ReporterData &	actions,
		const ReporterData &	log_probabilities,
		const std::vector< std::vector< Real > > &	rewards
	)		const

Assemble one trajectory per sample that holds at least one transition.

Parameters

observations	Observation reporter data.
actions	Action reporter data.
log_probabilities	Log-probability reporter data, one entry per action.
rewards	Reward reporter data indexed as [sample][raw time].

Returns

Assembled trajectories in sample order.

Definition at line 35 of file LibtorchRLTrajectoryAssembler.C.

{
  if (actions.size() != log_probabilities.size())
    mooseError("The number of log-probability reporters (",
               log_probabilities.size(),
               ") must match the number of action reporters (",
               actions.size(),
               ").");
 
  checkSampleCount(observations, "observation", rewards.size());
  checkSampleCount(actions, "action", rewards.size());
  checkSampleCount(log_probabilities, "log-probability", rewards.size());
 
  std::vector<LibtorchRLTrajectoryBuffer::Trajectory> trajectories;
 
  for (const auto sample_i : index_range(rewards))
  {
    const auto & reward_sample = rewards[sample_i];
    const auto num_transitions = numTransitions(reward_sample.size());
    if (!num_transitions)
      continue;
 
    std::vector<std::vector<Real>> component_trajectories(observations.size());
    for (const auto observation_i : index_range(observations))
      component_trajectories[observation_i] =
          downsample((*observations[observation_i])[sample_i], 0, num_transitions + 1);
 
    LibtorchRLTrajectoryBuffer::Trajectory trajectory;
    trajectory.observations.reserve(num_transitions);
    trajectory.next_observations.reserve(num_transitions);
    trajectory.actions.assign(num_transitions, std::vector<Real>());
    trajectory.log_probabilities.assign(num_transitions, std::vector<Real>());
 
    for (auto & action_row : trajectory.actions)
      action_row.reserve(actions.size());
    for (auto & log_probability_row : trajectory.log_probabilities)
      log_probability_row.reserve(log_probabilities.size());
 
    for (const auto step_i : make_range(num_transitions))
    {
      trajectory.observations.push_back(
          _observation_history.stackTrajectoryObservation(component_trajectories, step_i));
      trajectory.next_observations.push_back(
          _observation_history.stackTrajectoryObservation(component_trajectories, step_i + 1));
    }
 
    for (const auto action_i : index_range(actions))
    {
      const auto action_sequence =
          downsample((*actions[action_i])[sample_i], _shift_outputs, num_transitions);
      const auto log_probability_sequence =
          downsample((*log_probabilities[action_i])[sample_i], _shift_outputs, num_transitions);
 
      for (const auto step_i : make_range(num_transitions))
      {
        trajectory.actions[step_i].push_back(action_sequence[step_i]);
        trajectory.log_probabilities[step_i].push_back(log_probability_sequence[step_i]);
      }
    }
 
    trajectory.rewards = _average_reward_over_timestep_window
                             ? windowAverage(reward_sample, num_transitions)
                             : downsample(reward_sample, _timestep_window, num_transitions);
    // Full-solve rollout states are discarded after transfer, so GAE must stop at the boundary.
    trajectory.terminals.assign(num_transitions, false);
    trajectory.terminals.back() = true;
 
    trajectories.push_back(std::move(trajectory));
  }
 
  return trajectories;
}

Referenced by LibtorchDRLControlTrainer::collectTrajectoriesFromReporters().

checkSampleCount()

void LibtorchRLTrajectoryAssembler::checkSampleCount ( const ReporterData &  data, const std::string &  quantity, std::size_t  num_samples  )

staticprivate

Check that every component of a rollout quantity holds one sequence per sample.

Parameters

data	Reporter data to check.
quantity	Name of the rollout quantity used in the error message.
num_samples	Number of samples held by the reward reporter.

Definition at line 119 of file LibtorchRLTrajectoryAssembler.C.

{
  for (const auto component_i : index_range(data))
    if (data[component_i]->size() != num_samples)
      mooseError("Entry ",
                 component_i,
                 " of the ",
                 quantity,
                 " reporters holds ",
                 data[component_i]->size(),
                 " samples, but the reward reporter holds ",
                 num_samples,
                 " samples.");
}

Referenced by assemble().

downsample()

std::vector< Real > LibtorchRLTrajectoryAssembler::downsample ( const std::vector< Real > &  sample, unsigned int  offset, unsigned int  num_entries  ) const

private

Take every timestep_window-th entry of a raw reporter sequence.

Parameters

sample	Raw reporter sequence.
offset	Raw index of the first entry.
num_entries	Number of entries to extract.

Returns

Downsampled sequence.

Definition at line 137 of file LibtorchRLTrajectoryAssembler.C.

{
  std::vector<Real> values;
  values.reserve(num_entries);
 
  for (const auto entry_i : make_range(num_entries))
  {
    const auto raw_index = offset + entry_i * _timestep_window;
    if (raw_index >= sample.size())
      mooseError("Reporter data is shorter than required by the configured timestep window and "
                 "history stacking.");
    values.push_back(sample[raw_index]);
  }
 
  return values;
}

Referenced by assemble().

numTransitions()

unsigned int LibtorchRLTrajectoryAssembler::numTransitions

(

std::size_t

raw_sequence_size

)

const

Count the transitions available in a raw reporter sequence.

Parameters

raw_sequence_size

Number of raw time entries in the reporter sequence.

Returns

Number of complete timestep windows after the initial entry.

Definition at line 112 of file LibtorchRLTrajectoryAssembler.C.

{
  // Raw index 0 holds the initial value, so each transition consumes one full window after it.
  return raw_sequence_size ? (raw_sequence_size - 1) / _timestep_window : 0;
}

Referenced by assemble().

windowAverage()

std::vector< Real > LibtorchRLTrajectoryAssembler::windowAverage ( const std::vector< Real > &  sample, unsigned int  num_entries  ) const

private

Average a raw reporter sequence over consecutive timestep windows after the initial entry.

Parameters

sample	Raw reporter sequence.
num_entries	Number of windows to average.

Returns

Window-averaged sequence.

Definition at line 157 of file LibtorchRLTrajectoryAssembler.C.

{
  std::vector<Real> values;
  values.reserve(num_entries);
 
  for (const auto entry_i : make_range(num_entries))
  {
    const auto window_begin = 1 + entry_i * _timestep_window;
    const auto window_end = window_begin + _timestep_window;
    if (window_end > sample.size())
      mooseError(
          "Reporter reward data is shorter than required by the configured timestep window.");
 
    const Real sum =
        std::accumulate(sample.begin() + window_begin, sample.begin() + window_end, 0.0);
    values.push_back(sum / _timestep_window);
  }
 
  return values;
}

Referenced by assemble().

Member Data Documentation

_average_reward_over_timestep_window

const bool LibtorchRLTrajectoryAssembler::_average_reward_over_timestep_window

private

Whether rewards are averaged over each window instead of sampled at its end.

Definition at line 102 of file LibtorchRLTrajectoryAssembler.h.

Referenced by assemble().

_observation_history

const LibtorchObservationHistoryHelper LibtorchRLTrajectoryAssembler::_observation_history

private

Observation history stacking helper.

Definition at line 104 of file LibtorchRLTrajectoryAssembler.h.

Referenced by assemble().

_shift_outputs

const bool LibtorchRLTrajectoryAssembler::_shift_outputs

private

Whether actions and log probabilities are read one raw entry after the observation.

Definition at line 100 of file LibtorchRLTrajectoryAssembler.h.

Referenced by assemble().

_timestep_window

const unsigned int LibtorchRLTrajectoryAssembler::_timestep_window

private

Raw reporter entries per transition.

Definition at line 98 of file LibtorchRLTrajectoryAssembler.h.

Referenced by assemble(), downsample(), LibtorchRLTrajectoryAssembler(), numTransitions(), and windowAverage().

---

The documentation for this class was generated from the following files:

• stochastic_tools/include/libtorch/utils/LibtorchRLTrajectoryAssembler.h
• stochastic_tools/src/libtorch/utils/LibtorchRLTrajectoryAssembler.C