Conjugate Heat Transfer Simulation of a 67-Pebble Core
Contact: John Acierno (Penn State), Dillon Shaver (dshaver.at.anl.gov), and April Novak (anovak.at.anl.gov)
Model link: 67-Pebble Core Model
In this tutorial we are going to set up and simulate a simple Conjugate Heat Transfer (CHT) case using a helium (Pr=0.71) cooled 67-pebble bed. This tutorial was developed from an example case provided with NekRS and couples to MOOSE's CHT module using Cardinal as a wrapper. More information about the NEAMS tool Cardinal can be found on github, or on the Cardinal website. In each time step MOOSE will solve the energy equation in the the solid subdomain and pass the solution to NekRS, which will in-turn solve both the Navier-Stokes and energy equations in the fluid subdomain. NekRS will then pass its temperature solution back to MOOSE in the next time step. This transfer of information occurs at the boundary between solid and fluid subdomains, which are the pebble surfaces in this case.
Cardinal and MOOSE-Wrapped Apps
The analysis shown here is performed with Cardinal, a MOOSE application that "wraps" nekRS, a Computational Fluid Dynamics (CFD) code, and OpenMC, a Monte Carlo particle transport code. Cardinal is an open-source application, available on GitHub. As this example focuses on heat transfer modeling, there will be no further discussion of the OpenMC wrapping in Cardinal. "Wrapping" means that, for all intents and purposes, nekRS simulations can be run within the MOOSE framework and interacted with as if the physics and numerical solution performed by nekRS were a native MOOSE application. Cardinal contains source code that facilitates data transfers in and out of nekRS and runs nekRS within a MOOSE-controlled simulation. At a high level, Cardinal's wrapping of nekRS consists of:
Constructing a "mirror" of the nekRS mesh through which data transfers occur with MOOSE. For conjugate heat transfer applications such as those shown here, a MooseMesh is created by copying the nekRS surface mesh into a format that all native MOOSE applications can understand.
Adding MooseVariables to represent the nekRS solution. In other words, if nekRS stores the temperature internally as an
std::vector<double>
, with each entry corresponding to a nekRS node, then a MooseVariable is created that represents the same data, but that can be accessed in relation to the MooseMesh mirror.Writing multiphysics feedback fields in/out of nekRS's internal solution and boundary condition arrays. So, if nekRS represents a heat flux boundary condition internally as an
std::vector<double>
, this involves reading from a MooseVariable representing heat flux (which can be transferred with any of MOOSE's transfers to the nekRS wrapping) and writing into nekRS's internal vectors.
Accomplishing the above three tasks requires an intimate knowledge of how nekRS stores its solution fields and mesh. But once the wrapping is constructed, nekRS can then communicate with any other MOOSE application via the MultiApp and Transfer systems in MOOSE, enabling complex multiscale thermal-hydraulic analysis and multiphysics feedback. Note that the user does not need to construct this mapping themselves - it all happens automatically. The same wrapping can be used for conjugate heat transfer analysis with any MOOSE application that can compute a heat flux; that is, because a MOOSE-wrapped version of nekRS interacts with the MOOSE framework in a similar manner as natively-developed MOOSE applications, the agnostic formulations of the MultiApps and Transfers can be used to equally extract heat flux from Pronghorn, BISON, the MOOSE heat conduction module, and so on.
Cardinal allows nekRS and OpenMC to couple seamlessly with the MOOSE framework, enabling in-memory coupling, distributed parallel meshes for very large-scale applications, and straightforward multiphysics problem setup. Cardinal has capabilities for conjugate heat transfer coupling of nekRS and MOOSE, concurrent conjugate heat transfer and volumetric heat source coupling of nekRS and MOOSE, and volumetric density, temperature, and heat source coupling of OpenMC to MOOSE. Together, the OpenMC and nekRS wrappings augment MOOSE by expanding the framework to include high-resolution spectral element CFD and Monte Carlo particle transport. For conjugate heat transfer applications in particular, the libMesh-based interpolations of fields between meshes enables fluid-solid heat transfer simulations on meshes that are not necessarily continuous on phase interfaces, allowing mesh resolution to be specified based on the underlying physics, rather than rigid continuity restrictions in single-application heat transfer codes.
NekRS
nekRS solves the Navier-Stokes and energy conservation equations. They can be solved in either low-Mach or incompressible forms. This model uses the incompressible form,
(1)
where is the fluid velocity, is the fluid density, is the pressure, is the viscous stress tensor, is a body force vector, is a volumetric heat source in the fluid domain, is the fluid heat capacity, is the fluid temperature, and is the fluid thermal conductivity.
The form of the viscous stress tensor depends on the problem formulation and the selected turbulence model.
nekRS can be solved in either dimensional or non-dimensional form; in the present case, nekRS is solved in non-dimensional form. That is, characteristic scales for velocity, temperature, length, and time are defined and substituted into the governing equations so that all solution fields (velocity, pressure, temperature) are of order unity. A full derivation of the non-dimensional governing equations in nekRS is available here. To summarize, the non-dimensional quantities are defined as
where a superscript indicates a non-dimensional quantity and a "" subscript indicates a characteristic value. Inserting these definitions into Eq. (1) gives
(2)
Complete definitions of all terms are available here. New terms in these non-dimensional equations are and , the Reynolds and Peclet numbers, respectively:
nekRS solves for , , and . Cardinal will handle conversions from a non-dimensional solution to a dimensional MOOSE heat conduction application, but awareness of this non-dimensional formulation will be important for describing the nekRS input files.
This case uses an LES turbulence model in nekRS. For LES, the viscous stress tensor is given by
and a filtering operation is implemented as a source term to the momentum equation. More details about Nek filters can be found in Nek5000 Online Tutorial.
where is the filter strength and is an element-wise convolution operator that acts on the specified number of modes (usually two). These are integrated into the dimensionless momentum equation to yield
(3)
where may include other momentum source terms.
Heat Conduction Model
The MOOSE heat conduction module is used to solve for steady-state heat conduction,
(4)
where is the solid thermal conductivity, is the solid temperature, and is a volumetric heat source in the solid.
Geometry and computational model
The geometry for this case consists of 67 pebbles in a randomly packed cylindrical bed. The pebbles have a diameter of 1 (dimensionless) and the cylinder has a diameter of 4.4 (dimensionless). The pebble bed begins around 2.5 pebble diameters upstream from the cylinder inlet and has a total height of about 5 pebble diameters. A cross section of the fluid domain is shown in Figure 1 where the fluid is the gray region and the pebbles are the white regions.

Figure 1: A cross section of the fluid domain for the 67 pebble geometry.
NekRS Setup
For this case, the mesh file is provided through github Large File Storage (LFS) system. It can be downloaded in the following way
The mesh includes over 110,000 hexahedral elements and has been generated using a Voronoi cell approach Lan et al. (2021).
oudf file
We will begin by setting up the .oudf file. This file contains the setup for the required boundary conditions.
(htgr/pb67_cardinal/pb67.oudf)Note that we have bc->flux = bc->usrwrk[bc->idM]
in scalarNeumannConditions block. This line allows NekRS to use the heat flux provided by the MOOSE CHT module at the surface of the pebbles instead of using a constant heat flux as previously defined. There is also a kernel function defined in the .oudf file,
cliptOKL
. The
cliptOKL
kernel is used to limit extreme temperatures in the simulation which can occur in underresolved parts of the mesh. If the temperature is greater than 100 or less than 0, this kernel will set the temperature to 100 or 0 respectively.
udf file
The .udf file utilizes the kernal functions we defined in .oudf, and provides the detailed configurations of the related nekRS simulations.
(htgr/pb67_cardinal/pb67.udf)par file
The .par file lists the necessary input parameters to nekRS simulations.
(htgr/pb67_cardinal/pb67.par)If you want to run this example on GPUs, you simply need to pass --nekrs-backend CUDA --nekrs-device-id 0
on the command line later when we run the cardinal-opt
executable. Otherwise, this example will be run on a CPU backend. The specific filtering settings allow us to smooth out high frequencies. If you are planning on doing a DNS you should not include any filtering. Next, the parameters in [PRESSURE]
block are selected to have a short time-to-solution. We are able to achieve this by adding a preconditioner and a smoother. In NekRS version 21.1 the combination of semg preconditioner with
chebyshev+asm
smoothers works well, but in later versions 22.0+, using the amgx preconditioner with
chebyshev+jac
smoothers is much faster.
usr file
The .usr is a legacy from Nek5000 code, and we still use to specify the boundary conditons in Nek CFD calculations. Here we prescribe every element a boundaryID for both velocity and temperature. We are able to check the type of element using cbc(ifc,iel,1).eq.'TOP'
and assign boundary types based on that. In this case we have 5 types of elements: inlet, outlet, outer wall, pebble wall, and chamfer wall. Notice, for the chamfer we assign the wall condition for velocity, but the E condition for temperature. This results in flow moving around the chamfered area without an additional heat flux. The chamfer is necessary to prevent a mesh singularity between touching pebbles.
MOOSE setup
As previously stated, we strongly recommend completing the CHT tutorials on Cardinal, which fully describe every aspect of the MOOSE setup. Here, we will only cover case specific necessities.
In this case we need to create two files: nek.i and moose.i pertaining to the NekRS and MOOSE parameters respectively.
Starting with the simpler nek.i which contains the following
(htgr/pb67_cardinal/nek.i)In the [Mesh]
block, MOOSE will create a copy of the Nek mesh at the given boundaries. In this case it is at the pebble surfaces. In the
[Problem]
block we define the name of corresponding NekRS files. Next, we want MOOSE to use the same time steps as Nek so we prescribe that in the
[Executioner]
block. In the
[Output]
block we tell MOOSE to output an exodus file of the shallow Nek copy every 1,000 time steps. This can be a helpful check at the beginning of the simulation to make sure you are using the correct boundaries. Finally, in the
[Postprocessor]
block we define what values we want MOOSE to calculate for us. Here we want the integral flux, min, max, and average temperature at the pebble surface.
Note that in this file, we are copying the non-dimensional nekRS solution into Cardinal. In this case, the coupled MOOSE heat conduction will also be represented in non-dimensional form. If we wanted to instead transform the non-dimensional nekRS solution into an equivalent dimensional form (which would allow us to represent the solid heat conduction with dimensional units), we'd simply set nondimensional = true
for NekRSProblem
and then set the characteristic scales, U_ref
, T_ref
, dT_ref
, L_ref
, rho_0
, and Cp_0
. For more information, see the NekRSProblem documentation.
To setup the MOOSE mesh, a text file containing the center points of each pebble is necessary. For the bed used in this case, the file can be obtained here.
(htgr/pb67_cardinal/moose.i)Notice in the [Mesh]
block we are using a file to generate the solid mesh. Make sure to include the sphere.e file in your working directory. We use this single sphere of radius 1 and duplicate it using
CombinerGenerator
and all of the coordinates of the 67 pebbles given in positions.txt. From here we scale down the pebbles to ensure that they are not touching in the solid mesh. We then call the heat conduction module in the [Kernels]
block ensuring MOOSE will solve the temperature in the solid mesh. Then we tell MOOSE to match the boundary conditions use in Nek in the
[BCs]
block. In the
[Transfers]
block we define what and how variables get transfered. We define a multi app and transfer the temperature from Nek to MOOSE and return the average flux and flux integral back to Nek. Built in post-processors are used to obtain these values. Finally, we define an aux kernel to model the avg flux. Notice in nek.i the boundary from nek is 4 and in moose.i the boundary of interest is 1. Be sure not to mix up the boundaries in the fluid and solid mesh.
Results
Below are 3D renderings done in VisIt of the velocity (left) and temperature (right) fields.

Figure 2: 3D volumetric renderings done in VisIt of the velocity (left) and the temperature (right) fields.
References
- Yu-Hsiang Lan, Paul Fischer, Elia Merzari, and Misun Min.
All-hex meshing strategies for densely packed spheres.
arXiv, 2021.
URL: https://arxiv.org/abs/2106.00196.[BibTeX]