Using SALAMANDER on INL HPC Systems
This guide will discuss how to use SALAMANDER in an Idaho National Laboratory (INL) High Performance Computing (HPC) environment. To install SALAMANDER on your own HPC system, you should start by reviewing the MOOSE HPC Cluster instructions for information on how to install MOOSE (and MOOSE-based applications like SALAMANDER) and working with your system administrator to get the appropriate software dependencies installed. Once this is done, proceed with the manual HPC installation instructions below in Step Three.
Step One: Get an INL HPC Account
To use SALAMANDER on INL HPC, you first need an HPC account. Information on direct access to HPC, or through the web-based HPC OnDemand service, can be found on the MOOSE website INL HPC Services page.
Step Two: Set Up the Right Environment
An optimized, pre-packaged environment is available for use on the Sawtooth and Lemhi supercomputers. When logged in, run:
module load use.moose moose-dev
A set of appropriate MOOSE dependency modules will be loaded for use. Note that these modules require activation each time you log in.
Step Three: Clone SALAMANDER
SALAMANDER is hosted on GitHub, and should be cloned directly from there using git. As in the MOOSE directions, it is recommended that users create a directory named "projects" to put all of your MOOSE-related work.
To clone SALAMANDER, run the following commands in Terminal:
mkdir ~/projects
cd ~/projects
git clone https://github.com/idaholab/SALAMANDER.git
cd SALAMANDER
git checkout main
This sequence of commands downloads SALAMANDER from the GitHub server and checks out the "main" code branch. There are two code branches available:
"main", which is the current most-tested version of SALAMANDER for general usage, and
"devel", which is intended for code development (and may be more regularly broken as changes occur in SALAMANDER and MOOSE).
Developers wishing to add new features should create a new branch for submission off of the current "devel" branch.
Step Four: Build and Test SALAMANDER
To compile SALAMANDER, first make sure that the HPC environment is activated from Step Two (and be sure to do this any time that a new HPC Terminal instance is opened):
module load use.moose moose-dev
Then navigate to the SALAMANDER clone directory and download the MOOSE, TMAP8, and Cardinal submodules:
cd ~/projects/SALAMANDER
git submodule update --init moose
git submodule update --init tmap8
git submodule update --init cardinal
Next, some Cardinal dependencies need to be downloaded:
cd ~/projects/SALAMANDER/cardinal
git submodule update --init --recursive contrib/openmc
git submodule update --init contrib/DAGMC
git submodule update --init contrib/moab
Double-Down (and Embree) enable mixed precision ray tracing within DAGMC. While turned off by default for now, it can be enabled by adding the following to the above commands:
git submodule update --init contrib/embree
git submodule update --init contrib/double-down
and simply running export ENABLE_DOUBLE_DOWN=yes in your terminal environment prior to compiling SALAMANDER.
To download OpenMC cross sections needed for OpenMC-based Cardinal runs, run:
cd ~/projects/SALAMANDER/cardinal
scripts/download-openmc-cross-sections.sh
and subsequently set the location of those cross sections in your environment:
export OPENMC_CROSS_SECTIONS=~/projects/SALAMANDER/cross_sections/cross_sections.xml
This variable must be set in your environment anytime you wish to run SALAMANDER input files that utilize the OpenMC functionality within Cardinal! This can be done either using export on the command line, or placing this command within a shell config file (.bashrc, .zshrc, etc.).
The copies of MOOSE, TMAP8, and Cardinal provided with SALAMANDER have been fully tested against the current SALAMANDER version, and is guaranteed to work with all current SALAMANDER tests.
TMAP8 is an optional dependency of SALAMANDER and can be disabled to speed up build times, if TMAP8 capabilities are not needed. This can be done by simply not downloading the TMAP8 submodule (i.e., not performing the git submodule update --init tmap8 command above.)
Cardinal is an optional dependency of SALAMANDER and can be disabled to speed up build times, if Cardinal capabilities are not needed. This can be done by simply not downloading the Cardinal submodule (i.e., not performing the git submodule update --init cardinal command as well as the Cardinal dependency download commands above.)
Once all dependencies have been downloaded, SALAMANDER can be compiled and tested:
make -j8
Once SALAMANDER has successfully been compiled, the following command runs the active SALAMANDER tests:
./run_tests -j8
The -j8 flag in the above commands signifies the number of processor cores used to build the code and run the tests. The number in that flag can be changed to the desired number of physical and virtual cores on the HPC system being used to build SALAMANDER. If SALAMANDER is working correctly, all active tests will pass. This indicates that SALAMANDER is ready to be used and further developed.
The number of cores for building and testing on a Sawtooth or Lemhi "head node" / "sign-in node" should be kept as low as possible to not take up excessive system resources. Please be courteous to fellow users!
Step Five: Keep SALAMANDER Updated
Installation
Step One: Install Conda MOOSE Environment
In order to install SALAMANDER, the MOOSE developer environment must be installed. The installation procedure depends on your operating system, so click on the MOOSE website link below that corresponds to your operating system/platform and follow the instructions until you are done with the step named "Install MOOSE" (note that you do not need to follow the steps from the section titled "Cloning MOOSE" and below). Then, return to this page and continue with Step Two.
Advanced manual installation instructions for this environment are available via the MOOSE website.
If an error or other issue is experienced while using the conda environment, please see the MOOSE troubleshooting guide for Conda
Those users & developers interested in running SALAMANDER on an HPC cluster environment can find more information about that user case in our Using SALAMANDER on INL HPC Systems instructions.
Step Two: Clone SALAMANDER
SALAMANDER is hosted on GitHub, and should be cloned directly from there using git. As in the MOOSE directions, it is recommended that users create a directory named "projects" to put all of your MOOSE-related work.
To clone SALAMANDER, run the following commands in Terminal:
mkdir ~/projects
cd ~/projects
git clone https://github.com/idaholab/SALAMANDER.git
cd SALAMANDER
git checkout main
This sequence of commands downloads SALAMANDER from the GitHub server and checks out the "main" code branch. There are two code branches available:
"main", which is the current most-tested version of SALAMANDER for general usage, and
"devel", which is intended for code development (and may be more regularly broken as changes occur in SALAMANDER and MOOSE).
Developers wishing to add new features should create a new branch for submission off of the current "devel" branch.
Step Three: Build and Test SALAMANDER
To compile SALAMANDER, first make sure that the conda MOOSE environment is activated (and be sure to do this any time that a new Terminal window is opened):
conda activate moose
Then navigate to the SALAMANDER clone directory and download the MOOSE, TMAP8, and Cardinal submodules:
cd ~/projects/SALAMANDER
git submodule update --init moose
git submodule update --init tmap8
git submodule update --init cardinal
Next, some Cardinal dependencies need to be downloaded:
cd ~/projects/SALAMANDER/cardinal
git submodule update --init --recursive contrib/openmc
git submodule update --init contrib/DAGMC
git submodule update --init contrib/moab
Double-Down (and Embree) enable mixed precision ray tracing within DAGMC. While turned off by default for now, it can be enabled by adding the following to the above commands:
git submodule update --init contrib/embree
git submodule update --init contrib/double-down
and simply running export ENABLE_DOUBLE_DOWN=yes in your terminal environment prior to compiling SALAMANDER.
To download OpenMC cross sections needed for OpenMC-based Cardinal runs, run:
cd ~/projects/SALAMANDER/cardinal
scripts/download-openmc-cross-sections.sh
and subsequently set the location of those cross sections in your environment:
export OPENMC_CROSS_SECTIONS=~/projects/SALAMANDER/cross_sections/cross_sections.xml
This variable must be set in your environment anytime you wish to run SALAMANDER input files that utilize the OpenMC functionality within Cardinal! This can be done either using export on the command line, or placing this command within a shell config file (.bashrc, .zshrc, etc.).
The copies of MOOSE, TMAP8, and Cardinal provided with SALAMANDER have been fully tested against the current SALAMANDER version, and is guaranteed to work with all current SALAMANDER tests.
TMAP8 is an optional dependency of SALAMANDER and can be disabled to speed up build times, if TMAP8 capabilities are not needed. This can be done by simply not downloading the TMAP8 submodule (i.e., not performing the git submodule update --init tmap8 command above.)
Cardinal is an optional dependency of SALAMANDER and can be disabled to speed up build times, if Cardinal capabilities are not needed. This can be done by simply not downloading the Cardinal submodule (i.e., not performing the git submodule update --init cardinal command as well as the Cardinal dependency download commands above.)
Once all dependencies have been downloaded, SALAMANDER can be compiled and tested:
cd ~/projects/SALAMANDER
make -j8
./run_tests -j8
The -j8 flag in the above commands signifies the number of processor cores used to build the code and run the tests. The number in that flag can be changed to the number of physical and virtual cores on the workstation being used to build SALAMANDER.
If the build of Cardinal third-party libraries OpenMC, DAGMC, and MOAB is too slow on your machine, you can attempt to pass the number of make cores using the MAKEFLAGS variable. This can do a better job enabling proper parallelism in nested make builds (as is the case with Cardinal dependencies). To use this variable, simply perform:
make MAKEFLAGS=-j8
instead of the make commands highlighted above.
If SALAMANDER is working correctly, all active tests will pass. This indicates that SALAMANDER is ready to be used and further developed.
Update SALAMANDER
SALAMANDER (and its underlying dependencies MOOSE, TMAP8, and Cardinal) is under heavy development and is updated on a continuous basis. Therefore, it is important that the local copy of SALAMANDER be periodically updated to obtain new capabilities, improvements, and bugfixes. Weekly updates are recommended as, at minimum, the MOOSE submodule within SALAMANDER is updated up to several times a week.
While the SALAMANDER source code should be manually updated, the HPC modules (unlike the MOOSE conda packages) do not need to be kept up-to-date by the end user/developer. These are updated automatically for you by the HPC system administrators.
To update your SALAMANDER repository as a SALAMANDER user, use the following commands, which provide to general users the content of the most stable branch (upstream/main):
cd ~/projects/SALAMANDER
git checkout main
git fetch upstream
git rebase upstream/main
git submodule update moose
git submodule update tmap8
git submodule update cardinal
cd cardinal
git submodule update --recursive contrib/openmc
git submodule update contrib/DAGMC
git submodule update contrib/moab
To update your SALAMANDER repository as a SALAMANDER developer who regularly makes modifications to the code, use the following commands, which provide developers with the devel branch:
cd ~/projects/SALAMANDER
git checkout devel
git fetch upstream
git rebase upstream/devel
git submodule update moose
git submodule update tmap8
git submodule update cardinal
cd cardinal
git submodule update --recursive contrib/openmc
git submodule update contrib/DAGMC
git submodule update contrib/moab
TMAP8 is an optional dependency of SALAMANDER and can be disabled to speed up build times, if TMAP8 capabilities are not needed. This can be done by simply not downloading the TMAP8 submodule (i.e., not performing the git submodule update tmap8 command above.)
Cardinal is an optional dependency of SALAMANDER and can be disabled to speed up build times, if Cardinal capabilities are not needed. This can be done by simply not downloading the Cardinal submodule (i.e., not performing the git submodule update cardinal command as well as the following Cardinal dependency update commands above.)
Both sets of instructions assume that your copy of SALAMANDER is stored in ~/projects and that the idaholab/SALAMANDER git remote is labeled upstream. Use git remote -v in the SALAMANDER repository location to check for this and change the commands as necessary. Finally, SALAMANDER can be re-compiled and re-tested.
make -j8
./run_tests -j8
Viewing Results Remotely
You can use HPC OnDemand to view a SALAMANDER Exodus results file remotely. First, access the HPC OnDemand Dashboard, select Interactive Apps, and then select Linux Desktop with Visualization. Next, select your cluster (such as Sawtooth), the amount of time you believe you need, and then click Launch.
It may take some time before your 'Visualization Job' becomes available. When it does, simply click on it, and you will be presented a graphical user interface (GUI) desktop within your web browser. From here, you can open visualization applications (such as ParaView), and open your results file.
To use ParaView, open a terminal by clicking Applications at the top left, then click Terminal Emulator. A terminal window will open. Enter the following commands:
module load paraview
paraview
ParaView should open. From here, you can select File, Open, and navigate to the directory containing your Exodus results file, and open it.
Troubleshooting
If issues are experienced in installation and testing, several resources are available:
SALAMANDER Issues Page for SALAMANDER bugs or feature requests.
SALAMANDER Discussion Forum for SALAMANDER issues and questions.
MOOSE FAQ page for common MOOSE issues.
MOOSE Discussion Forum for non-SALAMANDER issues and questions.