INL HPC Cluster

Containerization

The development environment is deployed using containerization, which is a deployment practice that bundles all application (i.e., MOOSE or a MOOSE-based app) dependencies need to run on any host. This means that applications built within this environment can be executed across all INL HPC clusters. For example, you can build an application within a compute node on Sawtooth and execute the same executable of the application (providing you follow the instructions that follow) on both Bitterroot and Lemhi.

Versioning

The version of the dependencies used within MOOSE (PETSc, libMesh, etc) are tied directly to a given version of MOOSE. Given a version of MOOSE (or a MOOSE-based application), we produce a version of the development environment that contains the correct version of these dependencies. Whenever you are compiling MOOSE or a MOOSE-based application, you should use the corresponding version of the development environment.

In order to determine the version of the development environment to use when building, run the following within the MOOSE directory:

./scripts/versioner.py moose-dev

where the output will be a hashed version:

e930b1d

In this case, the version of that should be used is e930b1d. This is the version that should be used for loading the containerized environment module that follows.

Loading the Environment

To access all MOOSE related modules, you must first load the use.moose module. The development container modules are then located within the moose-dev-MPI modules, where MPI denotes the variant of MPI is used. The version of these moose-dev-MPI modules used are the versions described by the versioner script explained in the Versioning section. Currently, the only deployed variant is openmpi, i.e. the development container module that should be used is moose-dev-openmpi.

With this, the containerized development environment is loaded with the following:

module load use.moose moose-dev-openmpi/e930b1d

Take special note of the version applied to the moose-dev-openmpi module, which is e930b1d. Again, this version comes from the versioner script as described in Versioning. It is very important that you use the specific version that is required by the application that you are building. You will receive warnings during the compile process if the version is not correct.

Building

The containerized environment needs to be entered in order to build or execute. For the purposes of this documentation, we will use the version e930b1d, although yours could be different!

To enter the containerized shell environment to proceed with building, run the following (note that here we are requesting the specific version of the environment that we need):

module load use.moose moose-dev-openmpi/e930b1d
moose-dev-shell

You are now within the versioned, containerized environment. The standard MOOSE dependencies (PETSc, libMesh, WASP, etc) are built and there is not a need to run any of the update-and-rebuild scripts within MOOSE. You must repeat the commands above whenever you build any code.

At this point, we have assumed that you already have the source code for your application available and are in the directory that contains that application. Build your application and test it as normal:

make -j 4
./run_tests -j 4

Running

How your application is ran is dependent on whether or not you are running across multiple hosts. In general, you are running across multiple hosts when you are running within a job on HPC. We have two different methods of execution that we will reference later:

• Shell: Enters an interactive shell within the container where you can run multiple commands within the environment, can only be used on a single host

• Execute: Runs a single command within the container, required for running on multiple hosts

When running on a single host (i.e., not in a HPC job), enter the container in a shell as done above and execute your application as normal:

module load use.moose moose-dev-openmpi/e930b1d
moose-dev-shell
mpiexec -n 2 /path/to/your/application-opt -i ...

When you are running on multiple hosts (i.e., in an HPC job) you must use the "execute" method to run your application as described above. This is done by appending the command moose-dev-exec before the path to the binary of your application. That is, within an HPC job submission script or within an HPC job interactive session, you will run your application with:

module load use.moose moose-dev-openmpi/e930b1d
mpiexec -n 2 moose-dev-exec /path/to/your/application-opt -i ...

Note that the "execute" method will also work on a single host.

OnDemand Execution

You can utilize HPC OnDemand to enter an interactive code server (similar to VSCode) window on a compute host within the same containerized development environment that was previously described.

Proceed to the "VSCode Container" IDE application HPC OnDemand. If you are within the internal INL network (on an INL device), you can access this application here. If you are an external (non-INL) user, you can access this application here.

On the HPC OnDemand application page described above, take note of the "Module" dropdown box. Here you should see as an option moose-dev-openmpi/e930b1d. This module listing should contain the same module that you would have loaded within the Building and Running steps described above. Note again that the specific version that you need may not necessarily be e930b1d, in which case you should reference the Versioning step to obtain the proper version given your application.

Set the other options on the application page as desired. We suggest the following:

• Cores: At least four, although you are free to request more as needed.

• Working directory: The root directory of your application source. When developing/running on HPC, you should use your personal scratch storage over your home directory storage as scratch offers better performance.

Next, launch the job using the Launch button and connect to the instance once it is running. It will sometimes take a minute or two for the connection to succeed.

Once you have entered the code server window (the window that looks similar to that of VSCode), you may execute within the container using a terminal window in the instance. While within an OnDemand code server instance, there is no need to execute any of the moose-dev-shell or moose-dev-exec commands as described in the Running step. The code server window is ran within the container, thus you can run commands normally, like:

make -j 6
mpiexec -n 6 /path/to/myapp-opt -i ...