Moose::Kokkos::FESystem Class Reference

The Kokkos FE system class. More...

#include <KokkosFESystem.h>

Inheritance diagram for Moose::Kokkos::FESystem:

Public Member Functions
	FESystem (SystemBase &system)
	Constructor for standalone use (pure FE simulations) More...
	FESystem (System &base, SystemBase &system)
	Constructor for mixed FE+FV simulations, sharing device memory with an existing System. More...
void	reinit ()
	Allocate the quadrature point vectors for active variable and tags and cache quadrature point values. More...
KOKKOS_FUNCTION const auto &	getCoupling (unsigned int var) const
	Get the list of off-diagonal coupled variable numbers of a variable. More...
KOKKOS_FUNCTION bool	hasNodalBCMatrixTag (dof_id_type dof, TagID tag) const
	Check whether a local DOF index is associated with a nodal BC for an extra matrix tag. More...
KOKKOS_FUNCTION unsigned int	getFETypeID (unsigned int var) const
	Get the FE type ID of a variable. More...
KOKKOS_FUNCTION dof_id_type	getNodeLocalDofIndex (ContiguousNodeID node, unsigned int i, unsigned int var) const
	Get the local DOF index of a variable for a node. More...
KOKKOS_FUNCTION dof_id_type	getNodeGlobalDofIndex (ContiguousNodeID node, unsigned int var) const
	Get the global DOF index of a variable for a node. More...
KOKKOS_FUNCTION bool	isNodalDefined (ContiguousNodeID node, unsigned int var) const
	Get whether a variable is defined on a node. More...
KOKKOS_FUNCTION ADReal	getVectorDofADValue (const dof_id_type dof, const TagID tag, const Real seed) const
	Get the DOF value of a tagged vector for automatic differentiation (AD) More...
KOKKOS_FUNCTION Real &	getVectorQpValue (const ElementInfo info, const dof_id_type qp, const unsigned int var, const TagID tag) const
	Get the quadrature point value of a variable from a tagged vector. More...
KOKKOS_FUNCTION ADReal	getVectorQpADValue (const ElementInfo info, const dof_id_type offset, const dof_id_type qp, const unsigned int var, const TagID tag, const Real seed) const
	Get the quadrature point value of a variable from a tagged vector for automatic differentiation (AD) More...
KOKKOS_FUNCTION Real3 &	getVectorQpGrad (const ElementInfo info, const dof_id_type qp, const unsigned int var, const TagID tag) const
	Get the quadrature point gradient of a variable from a tagged vector. More...
KOKKOS_FUNCTION ADReal3	getVectorQpADGrad (const ElementInfo info, const Real33 jacobian, const dof_id_type offset, const dof_id_type qp, const unsigned int var, const TagID tag, const Real seed) const
	Get the quadrature point gradient of a variable from a tagged vector for automatic differentiation (AD) More...
KOKKOS_FUNCTION Real	getVectorQpValueFace (const ElementInfo info, const unsigned int side, const unsigned int qp, const unsigned int var, const TagID tag) const
	Get the face quadrature point value of a variable from a tagged vector. More...
KOKKOS_FUNCTION ADReal	getVectorQpADValueFace (const ElementInfo info, const unsigned int side, const unsigned int qp, const unsigned int var, const TagID tag, const Real seed) const
	Get the face quadrature point value of a variable from a tagged vector for automatic differentiation (AD) More...
KOKKOS_FUNCTION Real3	getVectorQpGradFace (const ElementInfo info, const unsigned int side, const Real33 jacobian, const unsigned int qp, const unsigned int var, const TagID tag) const
	Get the face quadrature point gradient of a variable from a tagged vector. More...
KOKKOS_FUNCTION ADReal3	getVectorQpADGradFace (const ElementInfo info, const unsigned int side, const Real33 jacobian, const unsigned int qp, const unsigned int var, const TagID tag, const Real seed) const
	Get the face quadrature point gradient of a variable from a tagged vector for automatic differentiation (AD) More...
KOKKOS_FUNCTION void	operator() (const ThreadID tid) const
	Kokkos function for caching variable values on element quadrature points. More...
void	sync (const MemcpyType dir)
	Synchronize the active tagged vectors and matrices between host and device. More...
void	setActiveVariables (const std::set< MooseVariableFieldBase *> &vars)
	Set the active variables. More...
void	setActiveSolutionTags (const std::set< TagID > &tags)
	Set the active solution tags. More...
void	setActiveResidualTags (const std::set< TagID > &tags)
	Set the active residual tags. More...
void	setActiveMatrixTags (const std::set< TagID > &tags)
	Set the active matrix tags. More...
void	clearActiveVariables ()
	Clear the cached active variables. More...
void	clearActiveSolutionTags ()
	Clear the cached active solution tags. More...
void	clearActiveResidualTags ()
	Clear the cached active residual tags. More...
void	clearActiveMatrixTags ()
	Clear the cached active matrix tags. More...
const libMesh::DofMap &	getDofMap () const
	Get the libMesh DOF map. More...
const Parallel::Communicator &	getComm () const
	Get the libMesh communicator. More...
const Array< Array< dof_id_type > > &	getLocalCommList () const
	Get the list of local DOF indices to communicate. More...
const Array< Array< dof_id_type > > &	getGhostCommList () const
	Get the list of ghost DOF indices to communicate. More...
const Sparsity &	getSparsity () const
	Get the sparisty pattern data. More...
KOKKOS_FUNCTION bool	isVariableActive (unsigned int var, ContiguousSubdomainID subdomain) const
	Check whether a variable is active on a subdomain. More...
KOKKOS_FUNCTION bool	isResidualTagActive (TagID tag) const
	Check whether a residual tag is active. More...
KOKKOS_FUNCTION bool	isMatrixTagActive (TagID tag) const
	Check whether a matrix tag is active. More...
KOKKOS_FUNCTION dof_id_type	getNumLocalDofs () const
	Get the number of local DOFs. More...
KOKKOS_FUNCTION dof_id_type	getNumGhostDofs () const
	Get the number of ghost DOFs. More...
KOKKOS_FUNCTION dof_id_type	getElemLocalDofIndex (ContiguousElementID elem, unsigned int i, unsigned int var) const
	Get the local DOF index of a variable for an element. More...
KOKKOS_FUNCTION dof_id_type	getElemGlobalDofIndex (ContiguousElementID elem, unsigned int i, unsigned int var) const
	Get the global DOF index of a variable for an element. More...
KOKKOS_FUNCTION dof_id_type	localToGlobalDofIndex (dof_id_type dof) const
	Get the global DOF index of a local DOF index. More...
KOKKOS_FUNCTION Vector &	getVector (TagID tag) const
	Get a tagged Kokkos vector. More...
KOKKOS_FUNCTION Matrix &	getMatrix (TagID tag) const
	Get a tagged Kokkos matrix. More...
KOKKOS_FUNCTION Real &	getVectorDofValue (const dof_id_type dof, const TagID tag) const
	Get the DOF value of a tagged vector. More...
KOKKOS_FUNCTION Real &	getMatrixValue (dof_id_type row, dof_id_type col, TagID tag) const
	Get an entry from a tagged matrix. More...
PerfGraph &	perfGraph ()
	Get the PerfGraph. More...
KOKKOS_FUNCTION const Mesh &	kokkosMesh () const
	Get the const reference of the Kokkos mesh. More...
KOKKOS_FUNCTION const Assembly &	kokkosAssembly () const
	Get the const reference of the Kokkos assembly. More...
void	sync (const std::set< TagID > &tags, const MemcpyType dir)
	Synchronize the specified tagged vectors between host and device. More...
void	sync (const std::vector< TagID > &tags, const MemcpyType dir)
void	sync (const TagID tag, const MemcpyType dir)
SystemBase &	getSystem ()
	Get the MOOSE system. More...
const SystemBase &	getSystem () const

Static Public Member Functions
static InputParameters	validParams ()

Protected Member Functions
PerfID	registerTimedSection (const std::string &section_name, const unsigned int level) const
	Call to register a named section for timing. More...
PerfID	registerTimedSection (const std::string &section_name, const unsigned int level, const std::string &live_message, const bool print_dots=true) const
	Call to register a named section for timing. More...
std::string	timedSectionName (const std::string &section_name) const

Protected Attributes
SystemBase &	_system
	Reference of the MOOSE system. More...
const MooseMesh &	_mesh
	Reference of the MOOSE mesh. More...
const libMesh::DofMap &	_dof_map
	Reference of the libMesh DOF map. More...
const Parallel::Communicator &	_comm
	Reference of the libMesh communicator. More...
const unsigned int	_num_vars
	Number of variables. More...
const dof_id_type	_num_local_dofs
	Number of local DOFs. More...
const dof_id_type	_num_ghost_dofs
	Number of ghost DOFs. More...
Array< Array2D< dof_id_type > >	_local_elem_dof_index
	Local element DOF indices of each variable. More...
Array< dof_id_type >	_local_to_global_dof_index
	Map from local DOF index to global DOF index. More...
Array< unsigned int >	_max_dofs_per_elem
	Maximum number of DOFs per element for each variable. More...
Array2D< bool >	_var_subdomain_active
	Whether each variable is active on subdomains. More...
Array< unsigned int >	_active_variables
	List of active variable numbers. More...
MooseApp &	_pg_moose_app
	The MooseApp that owns the PerfGraph. More...
const std::string	_prefix
	A prefix to use for all sections. More...
Array< Vector >	_vectors
	Kokkos vectors and matrices on device. More...
Array< Matrix >	_matrices
Array< TagID >	_active_solution_tags
	List of active tags. More...
Array< TagID >	_active_residual_tags
Array< TagID >	_active_matrix_tags
Array< bool >	_residual_tag_active
	Flag whether each tag is active. More...
Array< bool >	_matrix_tag_active
Array< Array< dof_id_type > >	_local_comm_list
	List of DOFs to send and receive. More...
Array< Array< dof_id_type > >	_ghost_comm_list

Private Member Functions
void	setupVariables ()
	Setup variable data. More...
void	setupDofs ()
	Setup DOF data. More...
void	setupCoupling ()
	Setup coupling data between variables. More...
void	setupNodalBCDofs ()
	Mark the DOFs covered by nodal BCs. More...
void	getNodalBCDofs (const NodalBCBase *nbc, Array< bool > &dofs)
	Get the list of DOFs covered by a nodal BC. More...

Private Attributes
Thread	_thread
	Kokkos thread object. More...
Array< Array< dof_id_type > >	_local_node_dof_index
	Local nodal DOF indices of each variable. More...
Array< unsigned int >	_var_fe_types
	FE type ID of each variable. More...
Array< Array< unsigned int > >	_coupling
	Off-diagonal coupled variable numbers of each variable. More...
Array< Array< bool > >	_nbc_matrix_tag_dof
	Per-matrix-tag local-plus-ghost DOF masks for nodal BC coverage. More...
Array< Array2D< Array< Real > > >	_qp_solutions
	Cached elemental quadrature values and gradients. More...
Array< Array2D< Array< Real3 > > >	_qp_solutions_grad

Detailed Description

The Kokkos FE system class.

Each system in MOOSE with FE variables has a corresponding Kokkos FE system.

Definition at line 27 of file KokkosFESystem.h.

Constructor & Destructor Documentation

FESystem() [1/2]

Moose::Kokkos::FESystem::FESystem

(

SystemBase &

system

)

Constructor for standalone use (pure FE simulations)

Parameters

system

The associated MOOSE system

FESystem() [2/2]

Moose::Kokkos::FESystem::FESystem	(	System &	base,
		SystemBase &	system
	)

Constructor for mixed FE+FV simulations, sharing device memory with an existing System.

Parameters

base	The existing Kokkos System whose device memory to share
system	The associated MOOSE system

Member Function Documentation

clearActiveMatrixTags()

void Moose::Kokkos::System::clearActiveMatrixTags ( )

inlineinherited

Clear the cached active matrix tags.

Definition at line 118 of file KokkosSystem.h.

  {
    _active_matrix_tags.destroy();
    _matrix_tag_active = false;
  }

clearActiveResidualTags()

void Moose::Kokkos::System::clearActiveResidualTags ( )

inlineinherited

Clear the cached active residual tags.

Definition at line 109 of file KokkosSystem.h.

  {
    _active_residual_tags.destroy();
    _residual_tag_active = false;
  }

clearActiveSolutionTags()

void Moose::Kokkos::System::clearActiveSolutionTags ( )

inlineinherited

Clear the cached active solution tags.

Definition at line 104 of file KokkosSystem.h.

{ _active_solution_tags.destroy(); }

clearActiveVariables()

void Moose::Kokkos::System::clearActiveVariables ( )

inlineinherited

Clear the cached active variables.

Definition at line 99 of file KokkosSystem.h.

{ _active_variables.destroy(); }

getComm()

const Parallel::Communicator& Moose::Kokkos::System::getComm ( ) const

inlineinherited

Get the libMesh communicator.

Returns

The libMesh communicator

Definition at line 143 of file KokkosSystem.h.

{ return _comm; }

getCoupling()

KOKKOS_FUNCTION const auto& Moose::Kokkos::FESystem::getCoupling ( unsigned int  var ) const

inline

Get the list of off-diagonal coupled variable numbers of a variable.

Parameters

var	The variable number

Returns

The list of off-diagonal coupled variable numbers

Definition at line 56 of file KokkosFESystem.h.

{ return _coupling[var]; }

getDofMap()

const libMesh::DofMap& Moose::Kokkos::System::getDofMap ( ) const

inherited

Get the libMesh DOF map.

Returns

The libMesh DOF map

getElemGlobalDofIndex()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::System::getElemGlobalDofIndex ( ContiguousElementID  elem, unsigned int  i, unsigned int  var  ) const

inlineinherited

Get the global DOF index of a variable for an element.

Parameters

elem	The contiguous element ID
i	The element-local DOF index
var	The variable number

Returns

The global DOF index

Definition at line 221 of file KokkosSystem.h.

  {
    return _local_to_global_dof_index[_local_elem_dof_index[var](i, elem)];
  }

getElemLocalDofIndex()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::System::getElemLocalDofIndex ( ContiguousElementID  elem, unsigned int  i, unsigned int  var  ) const

inlineinherited

Get the local DOF index of a variable for an element.

Parameters

elem	The contiguous element ID
i	The element-local DOF index
var	The variable number

Returns

The local DOF index

Definition at line 207 of file KokkosSystem.h.

Referenced by getVectorQpADGrad(), getVectorQpADGradFace(), getVectorQpADValue(), getVectorQpADValueFace(), getVectorQpGradFace(), and getVectorQpValueFace().

  {
    return _local_elem_dof_index[var](i, elem);
  }

getFETypeID()

KOKKOS_FUNCTION unsigned int Moose::Kokkos::FESystem::getFETypeID ( unsigned int  var ) const

inline

Get the FE type ID of a variable.

Parameters

var	The variable number

Returns

The FE type ID

Definition at line 74 of file KokkosFESystem.h.

{ return _var_fe_types[var]; }

getGhostCommList()

const Array<Array<dof_id_type> >& Moose::Kokkos::System::getGhostCommList ( ) const

inlineinherited

Get the list of ghost DOF indices to communicate.

Returns

The list of ghost DOF indices to communicate

Definition at line 155 of file KokkosSystem.h.

{ return _ghost_comm_list; }

getLocalCommList()

const Array<Array<dof_id_type> >& Moose::Kokkos::System::getLocalCommList ( ) const

inlineinherited

Get the list of local DOF indices to communicate.

Returns

The list of local DOF indices to communicate

Definition at line 149 of file KokkosSystem.h.

{ return _local_comm_list; }

getMatrix()

KOKKOS_FUNCTION Matrix& Moose::Kokkos::System::getMatrix ( TagID  tag ) const

inlineinherited

Get a tagged Kokkos matrix.

Parameters

tag	The matrix tag

Returns

The Kokkos matrix

Definition at line 250 of file KokkosSystem.h.

{ return _matrices[tag]; }

getMatrixValue()

KOKKOS_FUNCTION Real& Moose::Kokkos::System::getMatrixValue ( dof_id_type  row, dof_id_type  col, TagID  tag  ) const

inlineinherited

Get an entry from a tagged matrix.

Parameters

row	The local row index
col	The global column index
tag	The matrix tag

Returns

The entry from the tagged matrix

Definition at line 270 of file KokkosSystem.h.

  {
    return _matrices[tag](row, col);
  }

getNodalBCDofs()

void Moose::Kokkos::FESystem::getNodalBCDofs ( const NodalBCBase *  nbc, Array< bool > &  dofs  )

private

Get the list of DOFs covered by a nodal BC.

Parameters

nbc	The Kokkos nodal BC object
dofs	Local-plus-ghost DOF mask; entries are set true for DOFs covered by the nodal BC

getNodeGlobalDofIndex()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::FESystem::getNodeGlobalDofIndex ( ContiguousNodeID  node, unsigned int  var  ) const

inline

Get the global DOF index of a variable for a node.

Parameters

node	The contiguous node ID
var	The variable number

Returns

The global DOF index

Definition at line 96 of file KokkosFESystem.h.

  {
    return _local_to_global_dof_index[_local_node_dof_index[var][node]];
  }

getNodeLocalDofIndex()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::FESystem::getNodeLocalDofIndex ( ContiguousNodeID  node, unsigned int  i, unsigned int  var  ) const

inline

Get the local DOF index of a variable for a node.

Parameters

node	The contiguous node ID
i	The node-local DOF index
var	The variable number

Returns

The local DOF index

Definition at line 83 of file KokkosFESystem.h.

  {
    return _local_node_dof_index[var][node] + i;
  }

getNumGhostDofs()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::System::getNumGhostDofs ( ) const

inlineinherited

Get the number of ghost DOFs.

Returns

The number of ghost DOFs

Definition at line 198 of file KokkosSystem.h.

{ return _num_ghost_dofs; }

getNumLocalDofs()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::System::getNumLocalDofs ( ) const

inlineinherited

Get the number of local DOFs.

Returns

The number of local DOFs

Definition at line 192 of file KokkosSystem.h.

{ return _num_local_dofs; }

getSparsity()

const Sparsity& Moose::Kokkos::System::getSparsity ( ) const

inlineinherited

Get the sparisty pattern data.

Returns

The sparisty pattern data

Definition at line 161 of file KokkosSystem.h.

{ return _sparsity; }

getSystem() [1/2]

SystemBase& Moose::Kokkos::System::getSystem ( )

inlineinherited

Get the MOOSE system.

Returns

The MOOSE system

Definition at line 129 of file KokkosSystem.h.

{ return _system; }

getSystem() [2/2]

const SystemBase& Moose::Kokkos::System::getSystem ( ) const

inlineinherited

Definition at line 130 of file KokkosSystem.h.

{ return _system; }

getVector()

KOKKOS_FUNCTION Vector& Moose::Kokkos::System::getVector ( TagID  tag ) const

inlineinherited

Get a tagged Kokkos vector.

Parameters

tag	The vector tag

Returns

The Kokkos vector

Definition at line 243 of file KokkosSystem.h.

{ return _vectors[tag]; }

getVectorDofADValue()

KOKKOS_FUNCTION ADReal Moose::Kokkos::FESystem::getVectorDofADValue ( const dof_id_type  dof, const TagID  tag, const Real  seed  ) const

inline

Get the DOF value of a tagged vector for automatic differentiation (AD)

Parameters

dof	The local DOF index
tag	The vector tag
seed	The derivative seed

Returns

The DOF AD value with optional seed derivative

Definition at line 327 of file KokkosFESystem.h.

Referenced by getVectorQpADGrad(), getVectorQpADGradFace(), getVectorQpADValue(), and getVectorQpADValueFace().

{
  ADReal value = _vectors[tag][dof];

  if (seed != 0)
    value.derivatives().insert(_local_to_global_dof_index[dof]) = seed;

  return value;
}

getVectorDofValue()

KOKKOS_FUNCTION Real& Moose::Kokkos::System::getVectorDofValue ( const dof_id_type  dof, const TagID  tag  ) const

inlineinherited

Get the DOF value of a tagged vector.

Parameters

dof	The local DOF index
tag	The vector tag

Returns

The DOF value

Definition at line 258 of file KokkosSystem.h.

Referenced by getVectorQpGradFace(), and getVectorQpValueFace().

  {
    return _vectors[tag][dof];
  }

getVectorQpADGrad()

KOKKOS_FUNCTION ADReal3 Moose::Kokkos::FESystem::getVectorQpADGrad ( const ElementInfo  info, const Real33  jacobian, const dof_id_type  offset, const dof_id_type  qp, const unsigned int  var, const TagID  tag, const Real  seed  ) const

inline

Get the quadrature point gradient of a variable from a tagged vector for automatic differentiation (AD)

Parameters

info	The element information object
jacobian	The inverse Jacobian matrix
offset	The element's offset into the subdomain-local flattened quadrature point index
qp	The local quadrature point index
var	The variable number
tag	The vector tag
seed	The derivative seed

Returns

The quadrature AD gradient

Definition at line 363 of file KokkosFESystem.h.

{
  ADReal3 grad;

  if (seed == 0)
    grad = getVectorQpGrad(info, offset + qp, var, tag);
  else
  {
    auto fe = _var_fe_types[var];
    auto n_dofs = kokkosAssembly().getNumDofs(info.type, fe);
    auto & grad_phi = kokkosAssembly().getGradPhi(info.subdomain, info.type, fe);

    for (unsigned int i = 0; i < n_dofs; ++i)
      grad += getVectorDofADValue(getElemLocalDofIndex(info.id, i, var), tag, seed) *
              (jacobian * grad_phi(i, qp));
  }

  return grad;
}

getVectorQpADGradFace()

KOKKOS_FUNCTION ADReal3 Moose::Kokkos::FESystem::getVectorQpADGradFace ( const ElementInfo  info, const unsigned int  side, const Real33  jacobian, const unsigned int  qp, const unsigned int  var, const TagID  tag, const Real  seed  ) const

inline

Get the face quadrature point gradient of a variable from a tagged vector for automatic differentiation (AD)

Parameters

info	The element information object
side	The side index
jacobian	The inverse Jacobian matrix
qp	The local quadrature point index
var	The variable number
tag	The vector tag
seed	The derivative seed

Returns

The face quadrature AD gradient

Definition at line 450 of file KokkosFESystem.h.

{
  auto fe = _var_fe_types[var];
  auto n_dofs = kokkosAssembly().getNumDofs(info.type, fe);
  auto & grad_phi = kokkosAssembly().getGradPhiFace(info.subdomain, info.type, fe)(side);

  ADReal3 grad = ADReal(0);

  for (unsigned int i = 0; i < n_dofs; ++i)
    grad += getVectorDofADValue(getElemLocalDofIndex(info.id, i, var), tag, seed) *
            (jacobian * grad_phi(i, qp));

  return grad;
}

getVectorQpADValue()

KOKKOS_FUNCTION ADReal Moose::Kokkos::FESystem::getVectorQpADValue ( const ElementInfo  info, const dof_id_type  offset, const dof_id_type  qp, const unsigned int  var, const TagID  tag, const Real  seed  ) const

inline

Get the quadrature point value of a variable from a tagged vector for automatic differentiation (AD)

Parameters

info	The element information object
offset	The element's offset into the subdomain-local flattened quadrature point index
qp	The local quadrature point index
var	The variable number
tag	The vector tag
seed	The derivative seed

Returns

The quadrature AD value

Definition at line 338 of file KokkosFESystem.h.

{
  ADReal value = 0;

  if (seed == 0)
    value = getVectorQpValue(info, offset + qp, var, tag);
  else
  {
    auto fe = _var_fe_types[var];
    auto n_dofs = kokkosAssembly().getNumDofs(info.type, fe);
    auto & phi = kokkosAssembly().getPhi(info.subdomain, info.type, fe);

    for (unsigned int i = 0; i < n_dofs; ++i)
      value += getVectorDofADValue(getElemLocalDofIndex(info.id, i, var), tag, seed) * phi(i, qp);
  }

  return value;
}

getVectorQpADValueFace()

KOKKOS_FUNCTION ADReal Moose::Kokkos::FESystem::getVectorQpADValueFace ( const ElementInfo  info, const unsigned int  side, const unsigned int  qp, const unsigned int  var, const TagID  tag, const Real  seed  ) const

inline

Get the face quadrature point value of a variable from a tagged vector for automatic differentiation (AD)

Parameters

info	The element information object
side	The side index
qp	The local quadrature point index
var	The variable number
tag	The vector tag
seed	The derivative seed

Returns

The face quadrature AD value

Definition at line 409 of file KokkosFESystem.h.

{
  auto fe = _var_fe_types[var];
  auto n_dofs = kokkosAssembly().getNumDofs(info.type, fe);
  auto & phi = kokkosAssembly().getPhiFace(info.subdomain, info.type, fe)(side);

  ADReal value = 0;

  for (unsigned int i = 0; i < n_dofs; ++i)
    value += getVectorDofADValue(getElemLocalDofIndex(info.id, i, var), tag, seed) * phi(i, qp);

  return value;
}

getVectorQpGrad()

KOKKOS_FUNCTION Real3& Moose::Kokkos::FESystem::getVectorQpGrad ( const ElementInfo  info, const dof_id_type  qp, const unsigned int  var, const TagID  tag  ) const

inline

Get the quadrature point gradient of a variable from a tagged vector.

Parameters

info	The element information object
qp	The subdomain-local flattened quadrature point index
var	The variable number
tag	The vector tag

Returns

The quadrature gradient

Definition at line 162 of file KokkosFESystem.h.

Referenced by getVectorQpADGrad().

  {
    return _qp_solutions_grad[tag](info.subdomain, var)[qp];
  }

getVectorQpGradFace()

KOKKOS_FUNCTION Real3 Moose::Kokkos::FESystem::getVectorQpGradFace ( const ElementInfo  info, const unsigned int  side, const Real33  jacobian, const unsigned int  qp, const unsigned int  var, const TagID  tag  ) const

inline

Get the face quadrature point gradient of a variable from a tagged vector.

Parameters

info	The element information object
side	The side index
jacobian	The inverse Jacobian matrix
qp	The local quadrature point index
var	The variable number
tag	The vector tag

Returns

The face quadrature gradient

Definition at line 429 of file KokkosFESystem.h.

{
  auto fe = _var_fe_types[var];
  auto n_dofs = kokkosAssembly().getNumDofs(info.type, fe);
  auto & grad_phi = kokkosAssembly().getGradPhiFace(info.subdomain, info.type, fe)(side);

  Real3 grad = 0;

  for (unsigned int i = 0; i < n_dofs; ++i)
    grad += getVectorDofValue(getElemLocalDofIndex(info.id, i, var), tag) *
            (jacobian * grad_phi(i, qp));

  return grad;
}

getVectorQpValue()

KOKKOS_FUNCTION Real& Moose::Kokkos::FESystem::getVectorQpValue ( const ElementInfo  info, const dof_id_type  qp, const unsigned int  var, const TagID  tag  ) const

inline

Get the quadrature point value of a variable from a tagged vector.

Parameters

info	The element information object
qp	The subdomain-local flattened quadrature point index
var	The variable number
tag	The vector tag

Returns

The quadrature value

Definition at line 130 of file KokkosFESystem.h.

Referenced by getVectorQpADValue().

  {
    return _qp_solutions[tag](info.subdomain, var)[qp];
  }

getVectorQpValueFace()

KOKKOS_FUNCTION Real Moose::Kokkos::FESystem::getVectorQpValueFace ( const ElementInfo  info, const unsigned int  side, const unsigned int  qp, const unsigned int  var, const TagID  tag  ) const

inline

Get the face quadrature point value of a variable from a tagged vector.

Parameters

info	The element information object
side	The side index
qp	The local quadrature point index
var	The variable number
tag	The vector tag

Returns

The face quadrature value

Definition at line 390 of file KokkosFESystem.h.

{
  auto fe = _var_fe_types[var];
  auto n_dofs = kokkosAssembly().getNumDofs(info.type, fe);
  auto & phi = kokkosAssembly().getPhiFace(info.subdomain, info.type, fe)(side);

  Real value = 0;

  for (unsigned int i = 0; i < n_dofs; ++i)
    value += getVectorDofValue(getElemLocalDofIndex(info.id, i, var), tag) * phi(i, qp);

  return value;
}

hasNodalBCMatrixTag()

KOKKOS_FUNCTION bool Moose::Kokkos::FESystem::hasNodalBCMatrixTag ( dof_id_type  dof, TagID  tag  ) const

inline

Check whether a local DOF index is associated with a nodal BC for an extra matrix tag.

Parameters

dof	The local DOF index
tag	The extra matrix tag

Returns

Whether the local DOF index is covered by a nodal BC

Definition at line 64 of file KokkosFESystem.h.

  {
    return _nbc_matrix_tag_dof[tag].isAlloc() && _nbc_matrix_tag_dof[tag][dof];
  }

isMatrixTagActive()

KOKKOS_FUNCTION bool Moose::Kokkos::System::isMatrixTagActive ( TagID  tag ) const

inlineinherited

Check whether a matrix tag is active.

Parameters

tag	The matrix tag

Returns

Whether the matrix tag is active

Definition at line 186 of file KokkosSystem.h.

{ return _matrix_tag_active[tag]; }

isNodalDefined()

KOKKOS_FUNCTION bool Moose::Kokkos::FESystem::isNodalDefined ( ContiguousNodeID  node, unsigned int  var  ) const

inline

Get whether a variable is defined on a node.

Parameters

node	The contiguous node ID
var	The variable number

Returns

Whether the variable is defined on the node

Definition at line 107 of file KokkosFESystem.h.

  {
    return _local_node_dof_index[var][node] != libMesh::DofObject::invalid_id;
  }

isResidualTagActive()

KOKKOS_FUNCTION bool Moose::Kokkos::System::isResidualTagActive ( TagID  tag ) const

inlineinherited

Check whether a residual tag is active.

Parameters

tag	The residual tag

Returns

Whether the residual tag is active

Definition at line 179 of file KokkosSystem.h.

{ return _residual_tag_active[tag]; }

isVariableActive()

KOKKOS_FUNCTION bool Moose::Kokkos::System::isVariableActive ( unsigned int  var, ContiguousSubdomainID  subdomain  ) const

inlineinherited

Check whether a variable is active on a subdomain.

Parameters

var	The variable number
subdomain	The contiguous subdomain ID

Returns

Whether the variable is active

Definition at line 169 of file KokkosSystem.h.

  {
    return _var_subdomain_active(var, subdomain);
  }

kokkosAssembly()

KOKKOS_FUNCTION const Assembly& Moose::Kokkos::AssemblyHolder::kokkosAssembly ( ) const

inlineinherited

Get the const reference of the Kokkos assembly.

Returns

The const reference of the Kokkos assembly depending on the architecture this function is being called on

Definition at line 569 of file KokkosAssembly.h.

Referenced by getVectorQpADGrad(), getVectorQpADGradFace(), getVectorQpADValue(), getVectorQpADValueFace(), getVectorQpGradFace(), getVectorQpValueFace(), Moose::Kokkos::ElementReducer::operator()(), Moose::Kokkos::SideReducer::operator()(), Moose::Kokkos::NodalReducer::operator()(), Moose::Kokkos::NodalUserObject::operator()(), Moose::Kokkos::SideUserObject::operator()(), Moose::Kokkos::ElementUserObject::operator()(), Moose::Kokkos::ADNodalBC::operator()(), Moose::Kokkos::ADKernel::operator()(), Moose::Kokkos::ADIntegratedBC::operator()(), Moose::Kokkos::AuxKernel::operator()(), Moose::Kokkos::Material::operator()(), Moose::Kokkos::NodalBC::operator()(), Moose::Kokkos::NodalKernel::operator()(), Moose::Kokkos::IntegratedBC::operator()(), and Moose::Kokkos::Kernel::operator()().

  {
    KOKKOS_IF_ON_HOST(return _assembly_host;)

    return _assembly_device;
  }

kokkosMesh()

KOKKOS_FUNCTION const Mesh& Moose::Kokkos::MeshHolder::kokkosMesh ( ) const

inlineinherited

Get the const reference of the Kokkos mesh.

Returns

The const reference of the Kokkos mesh depending on the architecture this function is being called on

Definition at line 651 of file KokkosMesh.h.

Referenced by Moose::Kokkos::Assembly::computePhysicalMap(), Moose::Kokkos::LinearFVElementalKernel::operator()(), Moose::Kokkos::LinearFVFluxKernel::operator()(), Moose::Kokkos::Kernel::operator()(), Moose::Kokkos::IntegratedBC::operator()(), Moose::Kokkos::LinearFVBoundaryCondition::operator()(), Moose::Kokkos::Datum::q_point(), and KokkosBoundNodalKernel::skipOnBoundary().

  {
    KOKKOS_IF_ON_HOST(
        if (!_mesh_host.initialized()) mooseError(
            "kokkosMesh() was called too early. Kokkos mesh is available after problem "
            "initialization. Override initialSetup() if you need to setup your object data "
            "using the Kokkos mesh.");

        return _mesh_host;)

    return _mesh_device;
  }

localToGlobalDofIndex()

KOKKOS_FUNCTION dof_id_type Moose::Kokkos::System::localToGlobalDofIndex ( dof_id_type  dof ) const

inlineinherited

Get the global DOF index of a local DOF index.

Parameters

dof	The local DOF index

Returns

The global DOF index

Definition at line 233 of file KokkosSystem.h.

  {
    return _local_to_global_dof_index[dof];
  }

operator()()

KOKKOS_FUNCTION void Moose::Kokkos::FESystem::operator()

(

const ThreadID

tid

)

const

Kokkos function for caching variable values on element quadrature points.

perfGraph()

PerfGraph & PerfGraphInterface::perfGraph ( )

inherited

Get the PerfGraph.

Definition at line 86 of file PerfGraphInterface.C.

Referenced by CommonOutputAction::act(), PerfGraphData::finalize(), PerfGraphReporter::finalize(), and PerfGraphOutput::output().

{
  return _pg_moose_app.perfGraph();
}

registerTimedSection() [1/2]

PerfID PerfGraphInterface::registerTimedSection ( const std::string &  section_name, const unsigned int  level  ) const

protectedinherited

Call to register a named section for timing.

Parameters

section_name	The name of the code section to be timed
level	The importance of the timer - lower is more important (0 will always come out)

Returns

The ID of the section - use when starting timing

Definition at line 61 of file PerfGraphInterface.C.

{
  const auto timed_section_name = timedSectionName(section_name);
  if (!moose::internal::getPerfGraphRegistry().sectionExists(timed_section_name))
    return moose::internal::getPerfGraphRegistry().registerSection(timed_section_name, level);
  else
    return moose::internal::getPerfGraphRegistry().sectionID(timed_section_name);
}

registerTimedSection() [2/2]

PerfID PerfGraphInterface::registerTimedSection ( const std::string &  section_name, const unsigned int  level, const std::string &  live_message, const bool  print_dots = true  ) const

protectedinherited

Call to register a named section for timing.

Parameters

section_name	The name of the code section to be timed
level	The importance of the timer - lower is more important (0 will always come out)
live_message	The message to be printed to the screen during execution
print_dots	Whether or not progress dots should be printed for this section

Returns

The ID of the section - use when starting timing

Definition at line 72 of file PerfGraphInterface.C.

{
  const auto timed_section_name = timedSectionName(section_name);
  if (!moose::internal::getPerfGraphRegistry().sectionExists(timed_section_name))
    return moose::internal::getPerfGraphRegistry().registerSection(
        timedSectionName(section_name), level, live_message, print_dots);
  else
    return moose::internal::getPerfGraphRegistry().sectionID(timed_section_name);
}

reinit()

void Moose::Kokkos::FESystem::reinit

(

)

Allocate the quadrature point vectors for active variable and tags and cache quadrature point values.

setActiveMatrixTags()

void Moose::Kokkos::System::setActiveMatrixTags ( const std::set< TagID > &  tags )

inherited

Set the active matrix tags.

Parameters

vars	The active matrix tags

setActiveResidualTags()

void Moose::Kokkos::System::setActiveResidualTags ( const std::set< TagID > &  tags )

inherited

Set the active residual tags.

Parameters

tags	The active residual tags

setActiveSolutionTags()

void Moose::Kokkos::System::setActiveSolutionTags ( const std::set< TagID > &  tags )

inherited

Set the active solution tags.

Parameters

tags	The active solution tags

setActiveVariables()

void Moose::Kokkos::System::setActiveVariables ( const std::set< MooseVariableFieldBase *> &  vars )

inherited

Set the active variables.

Parameters

vars	The active MOOSE variables

setupCoupling()

void Moose::Kokkos::FESystem::setupCoupling ( )

private

Setup coupling data between variables.

setupDofs()

void Moose::Kokkos::FESystem::setupDofs ( )

private

Setup DOF data.

setupNodalBCDofs()

void Moose::Kokkos::FESystem::setupNodalBCDofs ( )

private

Mark the DOFs covered by nodal BCs.

setupVariables()

void Moose::Kokkos::FESystem::setupVariables ( )

private

Setup variable data.

sync() [1/4]

void Moose::Kokkos::System::sync ( const MemcpyType  dir )

inherited

Synchronize the active tagged vectors and matrices between host and device.

Parameters

dir	Copy direction

sync() [2/4]

void Moose::Kokkos::System::sync ( const std::set< TagID > &  tags, const MemcpyType  dir  )

inherited

Synchronize the specified tagged vectors between host and device.

Parameters

tags	The vector tags
dir	Copy direction

sync() [3/4]

void Moose::Kokkos::System::sync ( const std::vector< TagID > &  tags, const MemcpyType  dir  )

inherited

sync() [4/4]

void Moose::Kokkos::System::sync ( const TagID  tag, const MemcpyType  dir  )

inherited

timedSectionName()

std::string PerfGraphInterface::timedSectionName ( const std::string &  section_name ) const

protectedinherited

Returns

The name of the timed section with the name section_name.

Optionally adds a prefix if one is defined.

Definition at line 55 of file PerfGraphInterface.C.

Referenced by PerfGraphInterface::registerTimedSection().

{
  return _prefix.empty() ? "" : (_prefix + "::") + section_name;
}

validParams()

InputParameters PerfGraphInterface::validParams ( )

staticinherited

Definition at line 16 of file PerfGraphInterface.C.

Referenced by Convergence::validParams().

{
  InputParameters params = emptyInputParameters();
  return params;
}

Member Data Documentation

_active_matrix_tags

Array<TagID> Moose::Kokkos::System::_active_matrix_tags

protectedinherited

Definition at line 352 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::clearActiveMatrixTags().

_active_residual_tags

Array<TagID> Moose::Kokkos::System::_active_residual_tags

protectedinherited

Definition at line 351 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::clearActiveResidualTags().

_active_solution_tags

Array<TagID> Moose::Kokkos::System::_active_solution_tags

protectedinherited

List of active tags.

Definition at line 350 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::clearActiveSolutionTags().

_active_variables

Array<unsigned int> Moose::Kokkos::System::_active_variables

protectedinherited

List of active variable numbers.

Definition at line 344 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::clearActiveVariables().

_comm

const Parallel::Communicator& Moose::Kokkos::System::_comm

protectedinherited

Reference of the libMesh communicator.

Definition at line 296 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getComm().

_coupling

Array<Array<unsigned int> > Moose::Kokkos::FESystem::_coupling

private

Off-diagonal coupled variable numbers of each variable.

Definition at line 316 of file KokkosFESystem.h.

Referenced by getCoupling().

_dof_map

const libMesh::DofMap& Moose::Kokkos::System::_dof_map

protectedinherited

Reference of the libMesh DOF map.

Definition at line 291 of file KokkosSystem.h.

_ghost_comm_list

Array<Array<dof_id_type> > Moose::Kokkos::System::_ghost_comm_list

protectedinherited

Definition at line 368 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getGhostCommList().

_local_comm_list

Array<Array<dof_id_type> > Moose::Kokkos::System::_local_comm_list

protectedinherited

List of DOFs to send and receive.

Definition at line 367 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getLocalCommList().

_local_elem_dof_index

Array<Array2D<dof_id_type> > Moose::Kokkos::System::_local_elem_dof_index

protectedinherited

Local element DOF indices of each variable.

Definition at line 324 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getElemGlobalDofIndex(), and Moose::Kokkos::System::getElemLocalDofIndex().

_local_node_dof_index

Array<Array<dof_id_type> > Moose::Kokkos::FESystem::_local_node_dof_index

private

Local nodal DOF indices of each variable.

Definition at line 306 of file KokkosFESystem.h.

Referenced by getNodeGlobalDofIndex(), getNodeLocalDofIndex(), and isNodalDefined().

_local_to_global_dof_index

Array<dof_id_type> Moose::Kokkos::System::_local_to_global_dof_index

protectedinherited

Map from local DOF index to global DOF index.

Definition at line 329 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getElemGlobalDofIndex(), getNodeGlobalDofIndex(), getVectorDofADValue(), and Moose::Kokkos::System::localToGlobalDofIndex().

_matrices

Array<Matrix> Moose::Kokkos::System::_matrices

protectedinherited

Definition at line 318 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getMatrix(), and Moose::Kokkos::System::getMatrixValue().

_matrix_tag_active

Array<bool> Moose::Kokkos::System::_matrix_tag_active

protectedinherited

Definition at line 360 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::clearActiveMatrixTags(), and Moose::Kokkos::System::isMatrixTagActive().

_max_dofs_per_elem

Array<unsigned int> Moose::Kokkos::System::_max_dofs_per_elem

protectedinherited

Maximum number of DOFs per element for each variable.

Definition at line 334 of file KokkosSystem.h.

_mesh

const MooseMesh& Moose::Kokkos::System::_mesh

protectedinherited

Reference of the MOOSE mesh.

Definition at line 286 of file KokkosSystem.h.

_nbc_matrix_tag_dof

Array<Array<bool> > Moose::Kokkos::FESystem::_nbc_matrix_tag_dof

private

Per-matrix-tag local-plus-ghost DOF masks for nodal BC coverage.

Definition at line 321 of file KokkosFESystem.h.

Referenced by hasNodalBCMatrixTag().

_num_ghost_dofs

const dof_id_type Moose::Kokkos::System::_num_ghost_dofs

protectedinherited

Number of ghost DOFs.

Definition at line 311 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getNumGhostDofs().

_num_local_dofs

const dof_id_type Moose::Kokkos::System::_num_local_dofs

protectedinherited

Number of local DOFs.

Definition at line 306 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getNumLocalDofs().

_num_vars

const unsigned int Moose::Kokkos::System::_num_vars

protectedinherited

Number of variables.

Definition at line 301 of file KokkosSystem.h.

_pg_moose_app

MooseApp& PerfGraphInterface::_pg_moose_app

protectedinherited

The MooseApp that owns the PerfGraph.

Definition at line 135 of file PerfGraphInterface.h.

Referenced by PerfGraphInterface::perfGraph().

_prefix

const std::string PerfGraphInterface::_prefix

protectedinherited

A prefix to use for all sections.

Definition at line 138 of file PerfGraphInterface.h.

Referenced by PerfGraphInterface::timedSectionName().

_qp_solutions

Array<Array2D<Array<Real> > > Moose::Kokkos::FESystem::_qp_solutions

private

Cached elemental quadrature values and gradients.

Definition at line 299 of file KokkosFESystem.h.

Referenced by getVectorQpValue().

_qp_solutions_grad

Array<Array2D<Array<Real3> > > Moose::Kokkos::FESystem::_qp_solutions_grad

private

Definition at line 300 of file KokkosFESystem.h.

Referenced by getVectorQpGrad().

_residual_tag_active

Array<bool> Moose::Kokkos::System::_residual_tag_active

protectedinherited

Flag whether each tag is active.

Definition at line 359 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::clearActiveResidualTags(), and Moose::Kokkos::System::isResidualTagActive().

_system

SystemBase& Moose::Kokkos::System::_system

protectedinherited

Reference of the MOOSE system.

Definition at line 281 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getSystem().

_thread

Thread Moose::Kokkos::FESystem::_thread

private

Kokkos thread object.

Definition at line 293 of file KokkosFESystem.h.

_var_fe_types

Array<unsigned int> Moose::Kokkos::FESystem::_var_fe_types

private

FE type ID of each variable.

Definition at line 311 of file KokkosFESystem.h.

Referenced by getFETypeID(), getVectorQpADGrad(), getVectorQpADGradFace(), getVectorQpADValue(), getVectorQpADValueFace(), getVectorQpGradFace(), and getVectorQpValueFace().

_var_subdomain_active

Array2D<bool> Moose::Kokkos::System::_var_subdomain_active

protectedinherited

Whether each variable is active on subdomains.

Definition at line 339 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::isVariableActive().

_vectors

Array<Vector> Moose::Kokkos::System::_vectors

protectedinherited

Kokkos vectors and matrices on device.

Definition at line 317 of file KokkosSystem.h.

Referenced by Moose::Kokkos::System::getVector(), getVectorDofADValue(), and Moose::Kokkos::System::getVectorDofValue().

---

The documentation for this class was generated from the following file:

• include/kokkos/systems/KokkosFESystem.h