Kokkos Materials System

Before reading this documentation, consider reading the following materials first for a better understanding of this documentation:

commentnote

Kokkos-MOOSE materials do not support automatic differention yet.

A Kokkos-MOOSE material can be created by subclassing Moose::Kokkos::Material. Note that it should now be registered with registerKokkosMaterial() instead of registerMooseObject(). The hook method for material property computation, which used to be:


virtual void computeQpProperties() override;

in the original MOOSE materials, is now defined as a inlined public method with the following signature:


template <typename Derived>
KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const;

Other than the hook method definition, the Kokkos-MOOSE materials have several notable differences with the original MOOSE materials in the handling of material properties. The material property producer and consumer methods now have the signatures of

  • declareKokkosProperty<type, dimension>(name, dims), and

  • getKokkosMaterialProperty<type, dimension, state>(name),

respectively. Unlike in the original MOOSE materials where the type can be any valid C++ type, Kokkos-MOOSE materials require it to be a trivial type. Namely, it should not contain any virtual functions, dynamically allocated member variables, user-defined constructors, and other member variables of non-trivial types. While the code will not prevent using non-trivial types, it is your responsibility to understand the implications of using them and to make them behave properly. For instance, using a class with a user-defined constructor will have to be manually initialized as it is not called automatically upon allocation. Same applies to a class with in-class member initialization, which is equivalent to having a user-defined constructor. Using a class with dynamic allocations will incur a significant performance hit and will break when it is used for stateful material properties.

Instead, the material properties in Kokkos-MOOSE can be multi-dimensional to partially support the needs for dynamically-sized material properties. The dimension is provided as the second template argument dimension, which has the default value of 0 (scalar). The size of each dimension is provied as a vector as the function argument dims. When a material property is declared by multiple materials, it should have the same dimension over the entire domain, while the size of each dimension can be different between subdomains. However, a material property declared by boundary-restricted materials should have identical dimension sizes over the entire domain.

The material properties are stored as an object of type Moose::Kokkos::MaterialProperty<type, dimension>. Note that any material property object should be stored as a concrete instance. The producer and consumer methods also return the material property objects as copies. The only containers that are allowed to hold material properties dynamically are Moose::Kokkos::Array and Moose::Kokkos::Map.

The material property values of a quadrature point is accessed via operator() of Moose::Kokkos::MaterialProperty with datum and qp as arguments. It creates and returns a temporary object of type Moose::Kokkos::MaterialPropertyValue<type, dimension> which is a thin object that retrives the property values of the current quadrature point. Consider storing this temporary object locally to avoid object creation overhead every time it is called. For scalar properties, the temporary object can directly be cast into the property data type and overloads operator= so that a value can be directly assigned to it. For multi-dimensional properties, it provides operator() with dimensional indices like Moose::Kokkos::Array. The following examples illustrate the usage of scalar and multi-dimensional material properties, respectively:

  • Scalar (Moose::Kokkos::MaterialProperty<unsigned int>)


// Store the value
unsigned int value1 = _property1(datum, qp);
// Store the temporary object
auto prop1 = _property1(datum, qp);
auto prop2 = _property2(datum, qp);

// Compute material property (all are equivalent)
_property2(datum, qp) = value1 + 1;
_property2(datum, qp) = prop1 + 1;
prop2 = value1 + 1;
prop2 = prop1 + 1;

  • Multi-dimensional (Moose::Kokkos::MaterialProperty<unsigned int, 3>)


// Store the temporary object
auto prop = _property(datum, qp);

// Compute material property
for (unsigned int i = 0; i < n1; ++i)
  for (unsigned int j = 0; j < n2; ++j)
    for (unsigned int k = 0; k < n3; ++k)
      prop(i, j, k) = i + j + k;

See the following source codes of KokkosGenericConstantMaterial for an example of a material:

Listing 1: The KokkosGenericConstantMaterial header file.


#include "KokkosMaterial.h"

class KokkosGenericConstantMaterial : public Moose::Kokkos::Material
{
public:
  static InputParameters validParams();

  KokkosGenericConstantMaterial(const InputParameters & parameters);

  template <typename Derived>
  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    for (unsigned int i = 0; i < _num_props; ++i)
    {
      auto prop = _props[i](datum, qp);
      prop = _prop_values[i];
    }
  }

protected:
  // Material property names
  const std::vector<std::string> & _prop_names;
  // GPU-accessible array of property values
  const Moose::Kokkos::Array<Real> _prop_values;
  // GPU-accessible array of Kokkos material properties
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _props;
  // Number of properties
  const unsigned int _num_props;
};
(framework/include/kokkos/materials/KokkosGenericConstantMaterial.h)

Listing 2: The KokkosGenericConstantMaterial source file.


#include "KokkosGenericConstantMaterial.h"

registerKokkosMaterial("MooseApp", KokkosGenericConstantMaterial);

InputParameters
KokkosGenericConstantMaterial::validParams()
{
  InputParameters params = Material::validParams();
  params.addClassDescription(
      "Declares material properties based on names and values prescribed by input parameters.");
  params.addRequiredParam<std::vector<std::string>>(
      "prop_names", "The names of the properties this material will have");
  params.addRequiredParam<std::vector<Real>>("prop_values",
                                             "The values associated with the named properties");
  params.declareControllable("prop_values");

  return params;
}

KokkosGenericConstantMaterial::KokkosGenericConstantMaterial(const InputParameters & parameters)
  : Material(parameters),
    _prop_names(getParam<std::vector<std::string>>("prop_names")),
    _prop_values(getParam<std::vector<Real>>("prop_values")),
    _num_props(_prop_names.size())
{
  unsigned int num_names = _prop_names.size();
  unsigned int num_values = _prop_values.size();

  if (num_names != num_values)
    paramError("prop_names", "Size must match the number of prop_values.");

  _props.create(_num_props);

  for (unsigned int i = 0; i < _num_props; ++i)
    _props[i] = declareKokkosProperty<Real>(_prop_names[i]);
}
(framework/src/kokkos/materials/KokkosGenericConstantMaterial.K)

Stateful Material Properties

Stateful material properties can be obtained by getKokkosMaterialPropertyOld<type, dimension>(name) or getKokkosMaterialPropertyOlder<type, dimension>(name), or by specifying the state number (0 for current, 1 for old, and 2 for older) explicitly as the third template argument state of getKokkosMaterialProperty<type, dimension, state>(name) which defaults to 0. Stateful material properties can be optionally initialized by defining the following inlined public hook method:


template <typename Derived>
KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const

See the following source codes of KokkosStatefulTest for an example of stateful material properties:

Listing 3: The KokkosStatefulTest header file.


#pragma once

#include "KokkosMaterial.h"

class KokkosStatefulTest : public Moose::Kokkos::Material
{
public:
  static InputParameters validParams();

  KokkosStatefulTest(const InputParameters & parameters);

  template <typename Derived>
  KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const
  {
    if (_coupled_val)
      for (unsigned int i = 0; i < _num_props; ++i)
        _properties[i](datum, qp) = _coupled_val(datum, qp);
    else
      for (unsigned int i = 0; i < _num_props; ++i)
        _properties[i](datum, qp) = _prop_values[i];
  }
  template <typename Derived>
  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    // Really Expensive Fibonacci sequence generator!
    for (unsigned int i = 0; i < _num_props; ++i)
      _properties[i](datum, qp) = _properties_old[i](datum, qp) + _properties_older[i](datum, qp);
  }

private:
  // optional coupled variable
  const Moose::Kokkos::VariableValue _coupled_val;

  std::vector<std::string> _prop_names;
  Moose::Kokkos::Array<Real> _prop_values;

  unsigned int _num_props;

  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties;
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties_old;
  Moose::Kokkos::Array<Moose::Kokkos::MaterialProperty<Real>> _properties_older;
};
(test/include/kokkos/materials/KokkosStatefulTest.h)

Listing 4: The KokkosStatefulTest source file.


#include "KokkosStatefulTest.h"

registerKokkosMaterial("MooseTestApp", KokkosStatefulTest);

InputParameters
KokkosStatefulTest::validParams()
{
  InputParameters params = Material::validParams();
  params.addParam<std::vector<std::string>>("prop_names",
                                            "The names of the properties this material will have");
  params.addParam<std::vector<Real>>("prop_values",
                                     "The values associated with the named properties");
  params.addCoupledVar("coupled", "Coupled Value to be used in initQpStatefulProperties()");
  return params;
}

KokkosStatefulTest::KokkosStatefulTest(const InputParameters & parameters)
  : Material(parameters),
    _coupled_val(isParamValid("coupled") ? kokkosCoupledValue("coupled") : kokkosZeroValue()),
    _prop_names(getParam<std::vector<std::string>>("prop_names")),
    _prop_values(getParam<std::vector<Real>>("prop_values"))
{
  unsigned int num_names = _prop_names.size();
  unsigned int num_values = _prop_values.size();

  if (num_names != num_values)
    mooseError("Number of prop_names must match the number of prop_values for KokkosStatefulTest "
               "material!");

  _num_props = num_names;

  _properties.create(num_names);
  _properties_old.create(num_names);
  _properties_older.create(num_names);

  for (unsigned int i = 0; i < _num_props; ++i)
  {
    _properties[i] = declareKokkosProperty<Real>(_prop_names[i]);
    _properties_old[i] = getKokkosMaterialPropertyOld<Real>(_prop_names[i]);
    _properties_older[i] = getKokkosMaterialPropertyOlder<Real>(_prop_names[i]);
  }
}
(test/src/kokkos/materials/KokkosStatefulTest.K)

On-Demand Properties

In Kokkos-MOOSE, all material property data are stored for each quadrature point, regardless of whether the material properties are stateful or not. This is due to the massive parallelization of GPU that prevents storing material properties only for a single element or face. If you are unsure whether a material property will be actually requested by another object, you can declare the property as an on-demand property with declareKokkosOnDemandProperty<type, dimension>(name, dims). The storage of an on-demand property is only allocated when there is any object consuming the property, aiding in reducing the memory usage and computational cost for unused material properties. When accessing an on-demand property in its declaring material, you should always check the validity of the property. See the following section on optional properties for more details.

Optional Properties

There is no special method and object for weakly coupled material properties in Kokkos-MOOSE. Instead, a material property object will simply evaluate to false when it is uninitialized or when it holds an on-demand material property not requested by any object. Namely, if (property) will evaluate to false for the above conditions. You can query the existence of a material property with hasKokkosMaterialProperty<type, dimension>(name) and optionally initialize the material property object to reproduce the same behavior with the optional material properties in the original MOOSE.

See the following source codes of KokkosVarCouplingMaterial for an example of optional material properties:

Listing 5: The KokkosVarCouplingMaterial header file.


#pragma once

#include "KokkosMaterial.h"

/**
 * A material that couples a variable
 */
class KokkosVarCouplingMaterial : public Moose::Kokkos::Material
{
public:
  static InputParameters validParams();

  KokkosVarCouplingMaterial(const InputParameters & parameters);

  template <typename Derived>
  KOKKOS_FUNCTION void initQpStatefulProperties(const unsigned int qp, Datum & datum) const
  {
    _coupled_prop(datum, qp) = _var(datum, qp);
  }
  template <typename Derived>
  KOKKOS_FUNCTION void computeQpProperties(const unsigned int qp, Datum & datum) const
  {
    // If "declare_old" is set, then just use it. The test associated is checking that
    // initQpStatefulProperties can use a coupledValue
    if (_coupled_prop_old)
      _coupled_prop(datum, qp) = _coupled_prop_old(datum, qp);
    else
      _coupled_prop(datum, qp) = _base + _coef * _var(datum, qp);
  }

protected:
  const Moose::Kokkos::VariableValue _var;
  Real _base;
  Real _coef;
  Moose::Kokkos::MaterialProperty<Real> _coupled_prop;
  Moose::Kokkos::MaterialProperty<Real> _coupled_prop_old;
};
(test/include/kokkos/materials/KokkosVarCouplingMaterial.h)

Listing 6: The KokkosVarCouplingMaterial source file.


#include "KokkosVarCouplingMaterial.h"

registerKokkosMaterial("MooseTestApp", KokkosVarCouplingMaterial);

InputParameters
KokkosVarCouplingMaterial::validParams()
{
  InputParameters params = Material::validParams();
  params.addRequiredCoupledVar("var", "The variable to be coupled in");
  params.addParam<Real>("base", 0.0, "The baseline of the property");
  params.addParam<Real>("coef", 1.0, "The linear coefficient of the coupled var");
  params.addParam<bool>(
      "declare_old", false, "When True the old value for the material property is declared.");
  params.addParam<TagName>("tag", Moose::SOLUTION_TAG, "The solution vector to be coupled in");
  params.addParam<bool>("use_tag",
                        true,
                        "Whether the coupled value should come from a tag. If false, then we use "
                        "an ordinary coupled value.");
  params.addParam<MaterialPropertyName>(
      "coupled_prop_name",
      "diffusion",
      "The name of the material property that this material declares.");
  return params;
}

KokkosVarCouplingMaterial::KokkosVarCouplingMaterial(const InputParameters & parameters)
  : Material(parameters),
    _var(getParam<bool>("use_tag") ? kokkosCoupledVectorTagValue("var", "tag")
                                   : kokkosCoupledValue("var")),
    _base(getParam<Real>("base")),
    _coef(getParam<Real>("coef")),
    _coupled_prop(declareKokkosProperty<Real>("coupled_prop_name"))
{
  if (getParam<bool>("declare_old"))
    _coupled_prop_old = getKokkosMaterialPropertyOld<Real>("coupled_prop_name");
}
(test/src/kokkos/materials/KokkosVarCouplingMaterial.K)

Constant Properties

By default, material properties are computed and stored for each quadrature point. However, material properties are often constant within an element or subdomain, or vary little in space so that they can be approximated as constant without significant loss of accuracy. In this case, you can set the constant_on parameter of a material to ELEMENT or SUBDOMAIN, which makes the material properties declared by the material only computed and stored on per-element or per-subdomain basis. This can help save both computational cost and memory usage. However, note that a material property declared by boundary-restricted materials should have identical constant_on option across the entire domain.

Material Property Output

Material property output is not supported by Kokkos-MOOSE yet.

Advanced Topics

Evaluation of Material Properties on Element Faces

Analogously to the original MOOSE, Kokkos-MOOSE also creates three copies of materials for element and face material properties, which are distinguished by the combination of boolean flags _bnd and _neighbor. For Kokkos-MOOSE, it is crucial to optimize your material by switching off the declaration and evaluation of material properties that are not used on faces, because of the full storage of material properties. You can save a considerable amount of memory as well as computing time by switching off unused material properties. You can also leverage on-demand material properties to achieve the same effect.

Unsupported Material Types

The following material types are not supported by Kokkos-MOOSE yet:

  • Functor materials

  • Interface materials

  • Discrete materials

Available Objects