Sockeye Binary Access with INL-HPC

Familiarize yourself with HPC OnDemand, and return here with an interactive shell for the machine you wish to run your application on. NCRC Application binaries are only available on Sawtooth, Bitterroot, and WindRiver. The status and load on each machine can be found at hpcweb/status.

Load Sockeye Environment

Logged into either Sawtooth, Bitterroot, or WindRiver, load the Sockeye environment:


module load use.moose sockeye-openmpi

At this point you should be able to run sockeye-opt. For this example, Sockeye's --help can be displayed by running the following command:


sockeye-opt --help

Create a Test Input File

To demonstrate the use of Sockeye and to verify that it is operating correctly, create a test input file. For example, the following input file models steady-state diffusion and works with any MOOSE-based application:

[Mesh<<<{"href": "../../syntax/Mesh/index.html"}>>>]
  type = GeneratedMesh
  dim = 2
  nx = 10
  ny = 10
[]

[Variables<<<{"href": "../../syntax/Variables/index.html"}>>>]
  [u]
  []
[]

[Kernels<<<{"href": "../../syntax/Kernels/index.html"}>>>]
  [diff]
    type = Diffusion<<<{"description": "The Laplacian operator ($-\\nabla \\cdot \\nabla u$), with the weak form of $(\\nabla \\phi_i, \\nabla u_h)$.", "href": "../../source/kernels/Diffusion.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
  []
[]

[BCs<<<{"href": "../../syntax/BCs/index.html"}>>>]
  [left]
    type = DirichletBC<<<{"description": "Imposes the essential boundary condition $u=g$, where $g$ is a constant, controllable value.", "href": "../../source/bcs/DirichletBC.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
    boundary<<<{"description": "The list of boundary IDs from the mesh where this object applies"}>>> = left
    value<<<{"description": "Value of the BC"}>>> = 0
  []
  [right]
    type = DirichletBC<<<{"description": "Imposes the essential boundary condition $u=g$, where $g$ is a constant, controllable value.", "href": "../../source/bcs/DirichletBC.html"}>>>
    variable<<<{"description": "The name of the variable that this residual object operates on"}>>> = u
    boundary<<<{"description": "The list of boundary IDs from the mesh where this object applies"}>>> = right
    value<<<{"description": "Value of the BC"}>>> = 1
  []
[]

[Executioner<<<{"href": "../../syntax/Executioner/index.html"}>>>]
  type = Steady
  solve_type = 'PJFNK'
  petsc_options_iname = '-pc_type'
  petsc_options_value = 'hypre'
[]

[Outputs<<<{"href": "../../syntax/Outputs/index.html"}>>>]
  exodus<<<{"description": "Output the results using the default settings for Exodus output."}>>> = true
[]

This file (which we name input.i in the examples below) can be created directly in your INL-HPC home directory with a terminal editor such as emacs or vim, or it can be created on your local machine and copied via scp or rsync to the remote machine.

note:Copying files from your local machine

Copying files from your local machine to an HPC cluster first requires that you follow the instructions in Remote Access Primer. This workflow is for advanced users, comfortable with modifying their SSH settings, and familiar with their terminal in general.

In either case, first create a location for the file. For example, the following creates a "testing" folder in your remote home directory:


mkdir ~/testing

Example commands you would use with a terminal opened on your machine to copy files to HPC:


scp /path/to/the/local/input.i <your hpc user id>@hpclogin.inl.gov:~/testing/input.i
# or
rsync /path/to/the/local/input.i <your hpc user id>@hpclogin.inl.gov:~/testing/input.i

Run an Input File

(1) Start an interactive job

Simulations should not be executed on the login nodes. While this example is extremely lightweight, we will start an interactive job to get you in the habit of doing so.

Start by requesting a single processor for one hour within the "moose" project queue:


srun --time=01:00:00 --nodes=1 --ntasks=1 --cpus-per-task=1 --wckey=moose --pty bash

This command will likely take a few seconds or even minutes to execute depending the activity on the machine. When it completes you should see a message similar to the following:


srun: Your job requested six hours or less. Adding to the short partition.
srun: job 3844808 queued and waiting for resources
srun: job 3844808 has been allocated resources

When running on an interactive node, you will need to load the application environment once more:


module load use.moose sockeye-openmpi

(2) Run the simulation

In the previous section a test input file was created in your INL-HPC home directory within the "testing" folder. MOOSE-based applications are designed to be executed from the location of the input file, so start by navigating to this location:


cd ~/testing

Next, execute the application with the -i argument followed by the input filename:


sockeye-opt -i input.i

note:Permission Denied?

The available applications such as Sockeye will depend on being granted access to the specific application. Please refer to INL HPC Services for information about requesting access and the different levels available.

(3) Viewing Results

You can use HPC OnDemand to view your results remotely. Head to HPC OnDemand Dashboard, and select 'Interactive Apps' dropdown menu, and then click 'Linux Desktop with Visualization'. Select your cluster (such as Sawtooth), the amount of time you believe you need, and then click Launch. It may take some time before your 'Visualization Job' becomes available. When it does, simply click on it, and you will be presented with a GUI Desktop within your web browser. From here, you can open visualization applications (such as Paraview), and open your results file.

To use Paraview, open a terminal by: Clicking Applications at the top left, then click Terminal Emulator. A terminal window will open. Enter the following commands:


module load paraview
paraview

Paraview should open. From here, you can select File, Open, and navigate to the testing folder you created earlier. You should see your results file listed (input_out.e). Double click this file and enjoy!

However, in many cases it is more desirable to view the results on your local machine. This is done by copying the results file from the remote machine to your local machine using scp or rsnyc.

note:Copying files from remote HPC machine to your machine

Copying files from an HPC cluster to your machine first requires that you follow the instructions: Remote Access Primer. This workflow is for advanced users, comfortable with modifying their SSH settings, and familiar with their terminal in general.

Example commands you would use with a terminal opened on your machine to copy files from HPC:


scp <your hpc user id>@hpclogin:~/testing/input_out.e /path/to/the/local/destination

Scheduling Jobs with Slurm

INL HPC uses the Slurm job scheduling system. This section will provide you with a basic example of using Slurm for scheduling jobs to run using INL resources. For detailed information regarding using Slurm on INL HPC please visit HPC Web.

In general, scheduling jobs requires two steps:

create a job script that describes the required resources and the commands to execute, and
submit the script to the scheduler.

(1) Create Slurm Script

Let's use the same example input file from above to create a simple script for scheduling. The top portion of the script provides Slurm directives that are passed to scheduler. Notice that these lines are simply the arguments that are being passed to the scheduler, as was done via the command line in the interactive job above.


#!/bin/bash
#SBATCH --job-name=example
#SBATCH --output=example.log%j
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=112
#SBATCH --time=5:00
#SBATCH --partition=short
#SBATCH --wckey=moose

cd $SLURM_SUBMIT_DIR

module load use.moose sockeye-openmpi

mpiexec -n $SLURM_NTASKS sockeye-opt -i input.i Mesh/uniform_refine=7

It is recommend that this script exists in the same location where the input file resides. This will ensure that the associated output from the simulation ends up in the same location. For this example the script is located in the testing folder: ~/testing/test.sh.

The Slurm directives in this example are as follows:

--job-name=example sets the job name, referred to in various job information output.
--output=example.log%j specifies to save the console output to a file example.log<jobid>, where <jobid> is the job ID.
--nodes=1 specifies a single node (but not necessarily all of the resources on it).
--ntasks-per-node=112 specifies how many tasks (number of MPI ranks) to use per node. Here 112 corresponds to the number of physical cores per node on Bitterroot or Windriver. Sawtooth has 48 physical cores per node.
--time=5:00 allocates 5 minutes of time for the simulation.
--partition=short specifies to use the short partition, which has a max time of 6 hours; general has a longer max time.
--wckey=moose specifies moose as the project code.

For more Slurm documentation, INL HPC cluster specifications and partition limits, and project codes lists, see HPC Web.

note:The number of cores differ on each machine

The below table displays the possible number of cores for each available cluster

Cluster	Login Nodes	NCPUS	NODES
WindRiver	`windriver1`, `windriver2`	`112`	`843`
Bitterroot	`bitterroot1`, `bitterroot2`	`112`	`336` for `short`; `276` for `general`
Sawtooth	`sawtooth1`, `sawtooth2`	`48`	`2052`

On each of these systems, the general partition/queue has a maximum of 168 hours per job, and the short queue has a maximum of 6 hours.

Additional information for each cluster can be found at the INL HPC website.

The second portion provides the commands to be executed. The cd $SLURM_SUBMIT_DIR changes to the directory where the job was submitted from (on some clusters, the new job shell starts in the user's home directory by default). After loading the required modules with the module load command, the application is run using mpiexec, specifying the number of MPI processes with the -n argument, followed by the application command and any of its arguments. In this example, we pass a mesh refinement option to Sockeye to make the example more suitable for 112 cores.

(2) Submit Script

Submitting the job to the scheduler is trivial. First, be sure to log in to one the desired machines login nodes (e.g., using HPC OnDemand to aquire an Interactive Shell). Change directories to the location of the script and associated input files and then use the sbatch command.


cd ~/testing
sbatch test.sh

Running this command will return the job ID, for example:


Submitted batch job 3844761

The squeue command can be used to check the status of the jobs.


squeue -u <username>

where <username> is your user name.

When the job completes the expected output files will be available in the same location as the input file (~/testing). Refer to (3) Viewing Results in the previous example for more information regarding visualization.

(test/tests/kernels/simple_diffusion/simple_diffusion.i)

[Mesh]
  type = GeneratedMesh
  dim = 2
  nx = 10
  ny = 10
[]

[Variables]
  [u]
  []
[]

[Kernels]
  [diff]
    type = Diffusion
    variable = u
  []
[]

[BCs]
  [left]
    type = DirichletBC
    variable = u
    boundary = left
    value = 0
  []
  [right]
    type = DirichletBC
    variable = u
    boundary = right
    value = 1
  []
[]

[Executioner]
  type = Steady
  solve_type = 'PJFNK'
  petsc_options_iname = '-pc_type'
  petsc_options_value = 'hypre'
[]

[Outputs]
  exodus = true
[]

Load Sockeye Environment
Create a Test Input File
Run an Input File
Scheduling Jobs with Slurm