Kokkos Materials System

Before reading this documentation, consider reading the following materials first for a better understanding of this documentation:

Materials System to understand the MOOSE material system,
Getting Started with Kokkos-MOOSE to understand the programming practices for Kokkos-MOOSE,
Kokkos Kernels System to understand the common design pattern of objects in Kokkos-MOOSE.

note

Kokkos-MOOSE materials do not support automatic differention yet.

A Kokkos-MOOSE material can be created by subclassing Moose::Kokkos::Material in the same Curiosuly Recurring Template Pattern (CRTP) described in the Kokkos Kernels System. The hook method for material property computation, which used to be:


virtual void computeQpProperties() override;

in the original MOOSE materials, is now defined as a public method with the following signature:


KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const;

Other than the hook method definition, the Kokkos-MOOSE materials have several notable differences with the original MOOSE materials in the handling of material properties. The material property producer and consumer methods now have the signatures of

declareKokkosProperty<type, dimension>(name, dims), and
getKokkosMaterialProperty<type, dimension, state>(name),

respectively. Unlike in the original MOOSE materials where the type can be any valid C++ type, Kokkos-MOOSE materials require it to be a trivial type. Namely, it should not contain any virtual functions, dynamically allocated member variables, user-defined constructors, and other member variables of non-trivial types. While the code will not prevent using non-trivial types, it is your responsibility to understand the implications of using them and to make them behave properly. For instance, using a class with a user-defined constructor will have to be manually initialized as it is not called automatically upon allocation. Same applies to a class with in-class member initialization, which is equivalent to having a user-defined constructor. Using a class with dynamic allocations will incur a significant performance hit and will break when it is used for stateful material properties.

Instead, the material properties in Kokkos-MOOSE can be multi-dimensional to partially support the needs for dynamically-sized material properties. The dimension is provided as the second template argument dimension, which has the default value of 0 (scalar) and can be up to 4. The size of each dimension is provied as a vector as the function argument dims. It requires a material property to have the same dimension and size at every quadrature point.

The material properties are stored as an object of type Moose::Kokkos::MaterialProperty<type, dimension>. Note that any material property object should be stored as a concrete instance. The producer and consumer methods also return the material property objects as copies. The only containers that are allowed to hold material properties dynamically are Moose::Kokkos::Array and Moose::Kokkos::Map.

The material property values of a quadrature point is accessed via operator() of Moose::Kokkos::MaterialProperty with datum and qp as arguments. It creates and returns a temporary object of type Moose::Kokkos::MaterialPropertyValue<type, dimension> which is a thin object that retrives the property values of the current quadrature point. Consider storing this temporary object locally to avoid object creation overhead every time it is called. For scalar properties, the temporary object can directly be cast into the property data type and overloads operator= so that a value can be directly assigned to it. For multi-dimensional properties, it provides operator() with dimensional indices like Moose::Kokkos::Array. The following examples illustrate the usage of scalar and multi-dimensional material properties, respectively:

Scalar (Moose::Kokkos::MaterialProperty<unsigned int>)


// Store the value
unsigned int value1 = _property1(datum, qp);
// Store the temporary object
auto prop1 = _property1(datum, qp);
auto prop2 = _property2(datum, qp);

// Compute material property (all are equivalent)
_property2(datum, qp) = value1 + 1;
_property2(datum, qp) = prop1 + 1;
prop2 = value1 + 1;
prop2 = prop1 + 1;

Multi-dimensional (Moose::Kokkos::MaterialProperty<unsigned int, 3>)


// Store the temporary object
auto prop = _property(datum, qp);

// Compute material property
for (unsigned int i = 0; i < n1; ++i)
  for (unsigned int j = 0; j < n2; ++j)
    for (unsigned int k = 0; k < n3; ++k)
      prop(i, j, k) = i + j + k;

See the following source codes of KokkosGenericConstantMaterial for an example of a material:

Listing 1: The KokkosGenericConstantMaterial header file.


#include "KokkosMaterial.h"

class KokkosGenericConstantMaterial final
  : public Moose::Kokkos::Material<KokkosGenericConstantMaterial>
{
public:
  static InputParameters validParams();

  KokkosGenericConstantMaterial(const InputParameters & parameters);

  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    for (unsigned int i = 0; i < _num_props; ++i)
    {
      auto prop = _props[i](datum, qp);
      prop = _prop_values[i];
    }
  }

protected:
  // Material property names
  const std::vector<std::string> & _prop_names;
  // GPU-accessible array of property values
  const Moose::Kokkos::Array<Real> _prop_values;
  // GPU-accessible array of Kokkos material properties
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _props;
  // Number of properties
  const unsigned int _num_props;
};

Listing 2: The KokkosGenericConstantMaterial source file.


#include "KokkosGenericConstantMaterial.h"

registerMooseObject("MooseApp", KokkosGenericConstantMaterial);

InputParameters
KokkosGenericConstantMaterial::validParams()
{
  InputParameters params = Material::validParams();
  params.addClassDescription(
      "Declares material properties based on names and values prescribed by input parameters.");
  params.addRequiredParam<std::vector<std::string>>(
      "prop_names", "The names of the properties this material will have");
  params.addRequiredParam<std::vector<Real>>("prop_values",
                                             "The values associated with the named properties");
  params.declareControllable("prop_values");

  return params;
}

KokkosGenericConstantMaterial::KokkosGenericConstantMaterial(const InputParameters & parameters)
  : Material(parameters),
    _prop_names(getParam<std::vector<std::string>>("prop_names")),
    _prop_values(getParam<std::vector<Real>>("prop_values")),
    _num_props(_prop_names.size())
{
  unsigned int num_names = _prop_names.size();
  unsigned int num_values = _prop_values.size();

  if (num_names != num_values)
    paramError("prop_names", "Size must match the number of prop_values.");

  _props.create(_num_props);

  for (unsigned int i = 0; i < _num_props; ++i)
    _props[i] = declareKokkosProperty<Real>(_prop_names[i]);
}

Stateful Material Properties

Stateful material properties can be obtained by getKokkosMaterialPropertyOld<type, dimension>(name) or getKokkosMaterialPropertyOlder<type, dimension>(name), or by specifying the state number (0 for current, 1 for old, and 2 for older) explicitly as the third template argument state of getKokkosMaterialProperty<type, dimension, state>(name) which defaults to 0. Stateful material properties can be optionally initialized by defining the following public hook method:


KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const

See the following source codes of KokkosStatefulTest for an example of stateful material properties:

Listing 3: The KokkosStatefulTest header file.


#pragma once

#include "KokkosMaterial.h"

class KokkosStatefulTest final : public Moose::Kokkos::Material<KokkosStatefulTest>
{
public:
  static InputParameters validParams();

  KokkosStatefulTest(const InputParameters & parameters);

  KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const
  {
    if (_coupled_val)
      for (unsigned int i = 0; i < _num_props; ++i)
        _properties[i](datum, qp) = _coupled_val(datum, qp);
    else
      for (unsigned int i = 0; i < _num_props; ++i)
        _properties[i](datum, qp) = _prop_values[i];
  }

  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    // Really Expensive Fibonacci sequence generator!
    for (unsigned int i = 0; i < _num_props; ++i)
      _properties[i](datum, qp) = _properties_old[i](datum, qp) + _properties_older[i](datum, qp);
  }

private:
  // optional coupled variable
  const Moose::Kokkos::VariableValue _coupled_val;

  std::vector<std::string> _prop_names;
  Moose::Kokkos::Array<Real> _prop_values;

  unsigned int _num_props;

  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties;
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties_old;
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties_older;
};

Listing 4: The KokkosStatefulTest source file.


#include "KokkosStatefulTest.h"

registerMooseObject("MooseTestApp", KokkosStatefulTest);

InputParameters
KokkosStatefulTest::validParams()
{
  InputParameters params = Material::validParams();
  params.addParam<std::vector<std::string>>("prop_names",
                                            "The names of the properties this material will have");
  params.addParam<std::vector<Real>>("prop_values",
                                     "The values associated with the named properties");
  params.addCoupledVar("coupled", "Coupled Value to be used in initQpStatefulProperties()");
  return params;
}

KokkosStatefulTest::KokkosStatefulTest(const InputParameters & parameters)
  : Material(parameters),
    _coupled_val(isParamValid("coupled") ? kokkosCoupledValue("coupled") : kokkosZeroValue()),
    _prop_names(getParam<std::vector<std::string>>("prop_names")),
    _prop_values(getParam<std::vector<Real>>("prop_values"))
{
  unsigned int num_names = _prop_names.size();
  unsigned int num_values = _prop_values.size();

  if (num_names != num_values)
    mooseError(
        "Number of prop_names must match the number of prop_values for StatefulTest material!");

  _num_props = num_names;

  _properties.create(num_names);
  _properties_old.create(num_names);
  _properties_older.create(num_names);

  for (unsigned int i = 0; i < _num_props; ++i)
  {
    _properties[i] = declareKokkosProperty<Real>(_prop_names[i]);
    _properties_old[i] = getKokkosMaterialPropertyOld<Real>(_prop_names[i]);
    _properties_older[i] = getKokkosMaterialPropertyOlder<Real>(_prop_names[i]);
  }
}

Optional Properties

There is no special method and object for weakly coupled material properties in Kokkos-MOOSE. Instead, if a material property object is left uninitialized, it will simply evaluate to false. You can query the existence of a material property with hasKokkosMaterialProperty<type, dimension>(name) and optionally initialize the material property object to reproduce the same behavior with the optional material properties in the original MOOSE.

See the following source codes of KokkosVarCouplingMaterial for an example of optional material properties:

Listing 5: The KokkosVarCouplingMaterial header file.


#pragma once

#include "KokkosMaterial.h"

/**
 * A material that couples a variable
 */
class KokkosVarCouplingMaterial final : public Moose::Kokkos::Material<KokkosVarCouplingMaterial>
{
public:
  static InputParameters validParams();

  KokkosVarCouplingMaterial(const InputParameters & parameters);

  KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const
  {
    _coupled_prop(datum, qp) = _var(datum, qp);
  }
  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    // If "declare_old" is set, then just use it. The test associated is checking that
    // initQpStatefulProperties can use a coupledValue
    if (_coupled_prop_old)
      _coupled_prop(datum, qp) = _coupled_prop_old(datum, qp);
    else
      _coupled_prop(datum, qp) = _base + _coef * _var(datum, qp);
  }

protected:
  const Moose::Kokkos::VariableValue _var;
  Real _base;
  Real _coef;
  Moose::Kokkos::MaterialProperty<Real> _coupled_prop;
  Moose::Kokkos::MaterialProperty<Real> _coupled_prop_old;
};

Listing 6: The KokkosVarCouplingMaterial source file.


#include "KokkosVarCouplingMaterial.h"

registerMooseObject("MooseTestApp", KokkosVarCouplingMaterial);

InputParameters
KokkosVarCouplingMaterial::validParams()
{
  InputParameters params = Material::validParams();
  params.addRequiredCoupledVar("var", "The variable to be coupled in");
  params.addParam<Real>("base", 0.0, "The baseline of the property");
  params.addParam<Real>("coef", 1.0, "The linear coefficient of the coupled var");
  params.addParam<bool>(
      "declare_old", false, "When True the old value for the material property is declared.");
  params.addParam<TagName>("tag", Moose::SOLUTION_TAG, "The solution vector to be coupled in");
  params.addParam<bool>("use_tag",
                        true,
                        "Whether the coupled value should come from a tag. If false, then we use "
                        "an ordinary coupled value.");
  params.addParam<MaterialPropertyName>(
      "coupled_prop_name",
      "diffusion",
      "The name of the material property that this material declares.");
  return params;
}

KokkosVarCouplingMaterial::KokkosVarCouplingMaterial(const InputParameters & parameters)
  : Material(parameters),
    _var(getParam<bool>("use_tag") ? kokkosCoupledVectorTagValue("var", "tag")
                                   : kokkosCoupledValue("var")),
    _base(getParam<Real>("base")),
    _coef(getParam<Real>("coef")),
    _coupled_prop(declareKokkosProperty<Real>("coupled_prop_name"))
{
  if (getParam<bool>("declare_old"))
    _coupled_prop_old = getKokkosMaterialPropertyOld<Real>("coupled_prop_name");
}

Material Property Output

Material property output is not supported by Kokkos-MOOSE yet.

Advanced Topics

Evaluation of Material Properties on Element Faces

Likewise the original MOOSE, Kokkos-MOOSE also creates three copies of materials for element and face material properties, which are distinguished by the combination of boolean flags _bnd and _neighbor. And it is crucial to optimize your material by switching off the declaration and evaluation of material properties that are not used on faces. In Kokkos-MOOSE, all material property data are stored for each quadrature point, regardless of whether the material properties are stateful or not. This is due to the massive parallelization of GPU that prevents evaluating materials element-by-element or face-by-face. Therefore, you can save a considerable amount of memory as well as computing time by switching off unused material properties.

Unsupported Material Types

The following material types are not supported by Kokkos-MOOSE yet:

Functor materials
Interface materials
Discrete materials

Available Objects

Moose App
KokkosGenericConstantMaterialDeclares material properties based on names and values prescribed by input parameters.
Heat Transfer App
KokkosHeatConductionMaterialGeneral-purpose material model for heat conduction

(framework/include/kokkos/materials/KokkosGenericConstantMaterial.h)

// This file is part of the MOOSE framework
// https://www.mooseframework.org
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#include "KokkosMaterial.h"

class KokkosGenericConstantMaterial final
  : public Moose::Kokkos::Material<KokkosGenericConstantMaterial>
{
public:
  static InputParameters validParams();

  KokkosGenericConstantMaterial(const InputParameters & parameters);

  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    for (unsigned int i = 0; i < _num_props; ++i)
    {
      auto prop = _props[i](datum, qp);
      prop = _prop_values[i];
    }
  }

protected:
  // Material property names
  const std::vector<std::string> & _prop_names;
  // GPU-accessible array of property values
  const Moose::Kokkos::Array<Real> _prop_values;
  // GPU-accessible array of Kokkos material properties
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _props;
  // Number of properties
  const unsigned int _num_props;
};

(framework/src/kokkos/materials/KokkosGenericConstantMaterial.K)

// This file is part of the MOOSE framework
// https://www.mooseframework.org
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#include "KokkosGenericConstantMaterial.h"

registerMooseObject("MooseApp", KokkosGenericConstantMaterial);

InputParameters
KokkosGenericConstantMaterial::validParams()
{
  InputParameters params = Material::validParams();
  params.addClassDescription(
      "Declares material properties based on names and values prescribed by input parameters.");
  params.addRequiredParam<std::vector<std::string>>(
      "prop_names", "The names of the properties this material will have");
  params.addRequiredParam<std::vector<Real>>("prop_values",
                                             "The values associated with the named properties");
  params.declareControllable("prop_values");

  return params;
}

KokkosGenericConstantMaterial::KokkosGenericConstantMaterial(const InputParameters & parameters)
  : Material(parameters),
    _prop_names(getParam<std::vector<std::string>>("prop_names")),
    _prop_values(getParam<std::vector<Real>>("prop_values")),
    _num_props(_prop_names.size())
{
  unsigned int num_names = _prop_names.size();
  unsigned int num_values = _prop_values.size();

  if (num_names != num_values)
    paramError("prop_names", "Size must match the number of prop_values.");

  _props.create(_num_props);

  for (unsigned int i = 0; i < _num_props; ++i)
    _props[i] = declareKokkosProperty<Real>(_prop_names[i]);
}

(test/include/kokkos/materials/KokkosStatefulTest.h)

// This file is part of the MOOSE framework
// https://mooseframework.inl.gov
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#pragma once

#include "KokkosMaterial.h"

class KokkosStatefulTest final : public Moose::Kokkos::Material<KokkosStatefulTest>
{
public:
  static InputParameters validParams();

  KokkosStatefulTest(const InputParameters & parameters);

  KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const
  {
    if (_coupled_val)
      for (unsigned int i = 0; i < _num_props; ++i)
        _properties[i](datum, qp) = _coupled_val(datum, qp);
    else
      for (unsigned int i = 0; i < _num_props; ++i)
        _properties[i](datum, qp) = _prop_values[i];
  }

  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    // Really Expensive Fibonacci sequence generator!
    for (unsigned int i = 0; i < _num_props; ++i)
      _properties[i](datum, qp) = _properties_old[i](datum, qp) + _properties_older[i](datum, qp);
  }

private:
  // optional coupled variable
  const Moose::Kokkos::VariableValue _coupled_val;

  std::vector<std::string> _prop_names;
  Moose::Kokkos::Array<Real> _prop_values;

  unsigned int _num_props;

  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties;
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties_old;
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties_older;
};

(test/src/kokkos/materials/KokkosStatefulTest.K)

// This file is part of the MOOSE framework
// https://mooseframework.inl.gov
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#include "KokkosStatefulTest.h"

registerMooseObject("MooseTestApp", KokkosStatefulTest);

InputParameters
KokkosStatefulTest::validParams()
{
  InputParameters params = Material::validParams();
  params.addParam<std::vector<std::string>>("prop_names",
                                            "The names of the properties this material will have");
  params.addParam<std::vector<Real>>("prop_values",
                                     "The values associated with the named properties");
  params.addCoupledVar("coupled", "Coupled Value to be used in initQpStatefulProperties()");
  return params;
}

KokkosStatefulTest::KokkosStatefulTest(const InputParameters & parameters)
  : Material(parameters),
    _coupled_val(isParamValid("coupled") ? kokkosCoupledValue("coupled") : kokkosZeroValue()),
    _prop_names(getParam<std::vector<std::string>>("prop_names")),
    _prop_values(getParam<std::vector<Real>>("prop_values"))
{
  unsigned int num_names = _prop_names.size();
  unsigned int num_values = _prop_values.size();

  if (num_names != num_values)
    mooseError(
        "Number of prop_names must match the number of prop_values for StatefulTest material!");

  _num_props = num_names;

  _properties.create(num_names);
  _properties_old.create(num_names);
  _properties_older.create(num_names);

  for (unsigned int i = 0; i < _num_props; ++i)
  {
    _properties[i] = declareKokkosProperty<Real>(_prop_names[i]);
    _properties_old[i] = getKokkosMaterialPropertyOld<Real>(_prop_names[i]);
    _properties_older[i] = getKokkosMaterialPropertyOlder<Real>(_prop_names[i]);
  }
}

(test/include/kokkos/materials/KokkosVarCouplingMaterial.h)

// This file is part of the MOOSE framework
// https://mooseframework.inl.gov
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#pragma once

#include "KokkosMaterial.h"

/**
 * A material that couples a variable
 */
class KokkosVarCouplingMaterial final : public Moose::Kokkos::Material<KokkosVarCouplingMaterial>
{
public:
  static InputParameters validParams();

  KokkosVarCouplingMaterial(const InputParameters & parameters);

  KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const
  {
    _coupled_prop(datum, qp) = _var(datum, qp);
  }
  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    // If "declare_old" is set, then just use it. The test associated is checking that
    // initQpStatefulProperties can use a coupledValue
    if (_coupled_prop_old)
      _coupled_prop(datum, qp) = _coupled_prop_old(datum, qp);
    else
      _coupled_prop(datum, qp) = _base + _coef * _var(datum, qp);
  }

protected:
  const Moose::Kokkos::VariableValue _var;
  Real _base;
  Real _coef;
  Moose::Kokkos::MaterialProperty<Real> _coupled_prop;
  Moose::Kokkos::MaterialProperty<Real> _coupled_prop_old;
};

(test/src/kokkos/materials/KokkosVarCouplingMaterial.K)

// This file is part of the MOOSE framework
// https://mooseframework.inl.gov
//
// All rights reserved, see COPYRIGHT for full restrictions
// https://github.com/idaholab/moose/blob/master/COPYRIGHT
//
// Licensed under LGPL 2.1, please see LICENSE for details
// https://www.gnu.org/licenses/lgpl-2.1.html

#include "KokkosVarCouplingMaterial.h"

registerMooseObject("MooseTestApp", KokkosVarCouplingMaterial);

InputParameters
KokkosVarCouplingMaterial::validParams()
{
  InputParameters params = Material::validParams();
  params.addRequiredCoupledVar("var", "The variable to be coupled in");
  params.addParam<Real>("base", 0.0, "The baseline of the property");
  params.addParam<Real>("coef", 1.0, "The linear coefficient of the coupled var");
  params.addParam<bool>(
      "declare_old", false, "When True the old value for the material property is declared.");
  params.addParam<TagName>("tag", Moose::SOLUTION_TAG, "The solution vector to be coupled in");
  params.addParam<bool>("use_tag",
                        true,
                        "Whether the coupled value should come from a tag. If false, then we use "
                        "an ordinary coupled value.");
  params.addParam<MaterialPropertyName>(
      "coupled_prop_name",
      "diffusion",
      "The name of the material property that this material declares.");
  return params;
}

KokkosVarCouplingMaterial::KokkosVarCouplingMaterial(const InputParameters & parameters)
  : Material(parameters),
    _var(getParam<bool>("use_tag") ? kokkosCoupledVectorTagValue("var", "tag")
                                   : kokkosCoupledValue("var")),
    _base(getParam<Real>("base")),
    _coef(getParam<Real>("coef")),
    _coupled_prop(declareKokkosProperty<Real>("coupled_prop_name"))
{
  if (getParam<bool>("declare_old"))
    _coupled_prop_old = getKokkosMaterialPropertyOld<Real>("coupled_prop_name");
}

Stateful Material Properties
Optional Properties
Material Property Output
Advanced Topics
Available Objects