Submission and Job Information
The recommended way for Level 1 users to submit jobs to INL HPC is through an OnDemand form prepared for this purpose. The sections on this page provide details on how to properly fill in a submission form and submit a job. Figure 1 below provides a preview of the OnDemand interface with the NCRC menu item selected. Users will only have options to select codes for which they have a license.
Figure 1: NCRC OnDemand Landing Page
Specifying a Project
Every job submission is required to have a project specified. This is normally entered with the PBS "-P" option. However, you do not need to enter "-P" here, just the project name. Project names must be selected from the list on HPC Web. If more than one project name is appropriate, use your best judgment to choose the most applicable project name. If no project name is available that is applicable to your job, use one of the general-purpose technical area job names, e.g., “ne_gen” for nuclear energy applications (see Figure 2).
Figure 2: Top part of the NCRC job submission form.
Input File
This is the full path to your input file that will be submitted to the applicable executable. If your input file were to be submitted via the command line, it may look like:
You do not need the sockeye-opt -i and you need the full path to your equivalent of input.i. The "Select File" button should open a dialog that will allow you to browse your INL HPC folder structure and manually select your input file to prevent typos in the full path to the input file (see Figure 2).
Advanced Code and Input Settings
Advanced Parameters
By default, your submission of this form creates a basic submission command that gets submitted to the requested cluster.
This form field allows you to add additional parameters. For example, if you wish to run performance logging, you can add a -t or --timing in this box and your application would then be executed as:
You are allowed to put more than one in this box. Everything that you put there will be entered between sockeye-opt and your input file -i /home/user/input.i.
To get a full list and more explanation on these command line options, please see the command line usage documentation.
Specific Version Hash
This option allows you to run a specific build of the herd code. To get a list of the different version hashes that are allowed here, you can open a terminal on the appropriate cluster and run module spider sockeye and you will be presented with multiple versions of each herd application.
Working Directory Information
By default, your working directory is set to the same directory where your input file is located. You can change this if you would prefer a different directory by checking the checkbox and entering the new directory. You can also use the "Select Directory" button to get a file dialog box to select the directory instead of typing it out by hand.
Figure 3: Center part of the NCRC job submission form.
HPC Information
The HPC information section is an interactive form to create your scheduler submission.
Select the cluster you wish to run on. The amount of resources and the time you may request for your submission for will depend on the cluster.
Enter the number of hours for your request. If your submission finishes before the number of hours has passed, your job will end.
Enter the number of nodes for your job.
Select the number of cores for your job.
The following table shows the limitations of each system
| Cluster | Login Nodes | NCPUS | NODES | Max Hours |
|---|---|---|---|---|
| WindRiver | windriver1, windriver2 | 56 | 843 | 168 |
| Bitterroot | bitterroot1, bitterroot2 | 56 | 336 | 168 |
| Sawtooth | sawtooth1, sawtooth2 | 48 | 2052 | 168 |
Additional information for each cluster can be found at the INL HPC website.
Advanced HPC Submission Settings
By default, the scheduler decides how much memory to assign to the job if the amount of memory is not explicity provided. You can change this default behavior by clicking the "Show advanced HPC submission settings" checkbox and entering the amount of memory.
You can also select the max amount of memory for a node if you would like to use all of a node. Selecting all of the cores for a node also provides the max amount of memory.
Figure 4: Bottom part of the NCRC job submission form.