Using TMAP8 on INL HPC Systems
This guide will discuss how to use TMAP8 in an INL HPC environment. To install TMAP8 on your own HPC system, you should start by reviewing the MOOSE HPC Cluster instructions for information on how to install MOOSE (and MOOSE-based applications like TMAP8) and working with your system administrator to get the appropriate software dependencies installed. Once this is done, proceed with the manual HPC installation instructions below in Step Three.
Step One: Get an INL HPC Account
To use TMAP8 on INL HPC, you first need an HPC account. Information on direct access to HPC, or through the web-based HPC OnDemand service, can be found on the MOOSE website INL HPC Services page.
Step Two: Set Up the Right Environment
An optimized, pre-packaged environment is available for use on the Sawtooth and Lemhi supercomputers. When logged in, run:
module load use.moose moose-dev
A set of appropriate MOOSE dependency modules will be loaded for use. Note that these modules require activation each time you log in.
Step Three: Clone TMAP8
TMAP8 is hosted on GitHub, and should be cloned directly from there using git. As in the MOOSE directions, it is recommended that users create a directory named "projects" to put all of your MOOSE-related work.
To clone TMAP8, run the following commands in Terminal:
mkdir ~/projects
cd ~/projects
git clone https://github.com/idaholab/TMAP8.git
cd TMAP8
git checkout main
This sequence of commands downloads TMAP8 from the GitHub server and checks out the "main" code branch. There are two code branches available:
"main", which is the current most-tested version of TMAP8 for general usage, and
"devel", which is intended for code development (and may be more regularly broken as changes occur in TMAP8 and MOOSE).
Developers wishing to add new features should create a new branch for submission off of the current "devel" branch.
Step Four: Build and Test TMAP8
To compile TMAP8, first make sure that the HPC environment is activated from Step Two (and be sure to do this any time that a new HPC Terminal instance is opened):
module load use.moose moose-dev
Then navigate to the TMAP8 clone directory and download the MOOSE submodule:
cd ~/projects/TMAP8
git submodule update --init moose
To have access to the optional thermochemistry library Thermochimica Piro et al. (2013) provided within the MOOSE Chemical Reactions Module, check out the corresponding submodule by performing the following before building:
cd ~/projects/TMAP8/moose
git submodule update --init --checkout modules/chemical_reactions/contrib/thermochimica
The copy of MOOSE provided with TMAP8 has been fully tested against the current TMAP8 version, and is guaranteed to work with all current TMAP8 tests.
Once MOOSE is downloaded, several MOOSE dependencies need to be built: PETSc, libMesh, and WASP. These can be built easily using update-and-rebuild scripts within MOOSE. To speed up these build times, it is highly recommended to set the environment variable MOOSE_JOBS, which will speed up compilation time by using more than one core to build the dependency. One can set this variable to signal that the scripts use of 8 computer cores by doing the following:
export MOOSE_JOBS=4 METHODS=opt
Then, the dependencies can be built by running the installation scripts for MOOSE dependencies in the following sequence:
cd ~/projects/TMAP8/moose
scripts/update_and_rebuild_petsc.sh
scripts/update_and_rebuild_libmesh.sh
scripts/update_and_rebuild_wasp.sh
Once all dependencies have been downloaded and installed, TMAP8 can be compiled and tested:
cd ~/projects/TMAP8
make -j8
Once TMAP8 has successfully been compiled, the following command runs the active TMAP8 tests:
./run_tests -j8
The -j8 flag in the above commands signifies the number of processor cores used to build the code and run the tests. The number in that flag can be changed to the desired number of physical and virtual cores on the HPC system being used to build TMAP8. If TMAP8 is working correctly, all active tests will pass. This indicates that TMAP8 is ready to be used and further developed.
The number of cores for building and testing on a Sawtooth or Lemhi "head node" / "sign-in node" should be kept as low as possible to not take up excessive system resources. Please be courteous to fellow users!
Step Five: Keep TMAP8 Updated
TMAP8 (and the underlying MOOSE Framework) is under heavy development and is updated on a continuous basis. Therefore, it is important that the local copy of TMAP8 be periodically updated to obtain new capabilities, improvements, and bugfixes. Weekly updates are recommended as, at minimum, the MOOSE submodule within TMAP8 is updated up to several times a week.
While the TMAP8 source code should be manually updated, the HPC modules (unlike the MOOSE conda packages) do not need to be kept up-to-date by the end user/developer. These are updated automatically for you by the HPC system administrators.
To update your TMAP8 repository as a TMAP8 user, use the following commands, which provide to general users the content of the most stable branch (upstream/main):
cd ~/projects/TMAP8
git checkout main
git fetch upstream
git rebase upstream/main
git submodule update moose
To update your TMAP8 repository as a TMAP8 developer who regularly makes modifications to the code, use the following commands, which provide developers with the devel branch:
cd ~/projects/TMAP8
git checkout devel
git fetch upstream
git rebase upstream/devel
git submodule update moose
Both sets of instructions assume that your copy of TMAP8 is stored in ~/projects and that the idaholab/TMAP8 git remote is labeled upstream. Use git remote -v
If there were updates to MOOSE dependencies, the update-and-rebuild scripts should be re-run. It is only necessary to re-run those scripts for dependencies that were actually updated! It is usually helpful to check the most recent MOOSE Newsletter or Discussion Forum for any update announcements from the MOOSE development team for the preceding month.
export MOOSE_JOBS=4 METHODS=opt
cd ~/projects/TMAP8/moose
scripts/update_and_rebuild_petsc.sh
scripts/update_and_rebuild_libmesh.sh
scripts/update_and_rebuild_wasp.sh
Finally, TMAP8 can be re-compiled and re-tested. include-end=True
cd ~/projects/TMAP8
make -j8
./run_tests -j8
Viewing Results Remotely
You can use HPC OnDemand to view a TMAP8 Exodus results file remotely. First, access the HPC OnDemand Dashboard, select Interactive Apps, and then select Linux Desktop with Visualization. Next, select your cluster (such as Sawtooth), the amount of time you believe you need, and then click Launch.
It may take some time before your 'Visualization Job' becomes available. When it does, simply click on it, and you will be presented a graphical user interface (GUI) desktop within your web browser. From here, you can open visualization applications (such as ParaView), and open your results file.
To use ParaView, open a terminal by clicking Applications at the top left, then click Terminal Emulator. A terminal window will open. Enter the following commands:
module load paraview
paraview
ParaView should open. From here, you can select File, Open, and navigate to the directory containing your Exodus results file, and open it.
Troubleshooting
If issues are experienced in installation and testing, several resources are available:
TMAP8 Issues Page for TMAP8 bugs or feature requests.
TMAP8 Discussion Forum for TMAP8 issues and questions.
MOOSE FAQ page for common MOOSE issues.
MOOSE Discussion Forum for non-TMAP8 issues and questions.
References
- M.H.A. Piro, S. Simunovic, T.M. Besmann, B.J. Lewis, and W.T. Thompson.
The thermochemistry library thermochimica.
Computational Materials Science, 67:266–272, 2013.
URL: https://www.sciencedirect.com/science/article/pii/S0927025612005502, doi:https://doi.org/10.1016/j.commatsci.2012.09.011.[BibTeX]