Application Usage link

Physics and Syntax

Application Development link

Framework Development link

MOOSEDocs link

Infrastructure

Questions

Information and Tools

HPC Cluster

The following instructions aims at setting up a baseline single-user environment for building MOOSE based applications in a job scheduling capable environment.

If instead, you are here interested in allowing MOOSE-based development to be made avaialable to multiple users, please see our Multi-User setup instructions (requires administrative rights). Normal users: please ingore the previous sentence, and continue on...

Pre-Reqs

What ever compiler you choose to use on your cluster (GCC/Clang, MPICH/OpenMPI), the minimum requirement, is that it must be C++11 compatible. If you are unsure, please consult with your system admins for your cluster on which compiler to use (and how to use it).

  • CMake. A modern version of CMake (>2.8) is required to build some of the meta packages we need to include in PETSc.

  • Python 2.7.x Development libraries.

Your cluster will most likely have these two requirements available via some form of environment management software. If you are unfamiliar with how to manage your environment, please consult with your cluster administrators.

PREFIX Setup


export STACK_SRC=`mktemp -d /tmp/stack_temp.XXXXXX`
export PACKAGES_DIR=$HOME/moose-compilers
note

  • The $PACKAGES_DIR directory must reside in a location where all the compute nodes can access.

  • We need the above two environment variables to exist throughout these instructions. Please use the one and only one, terminal you executed those two commands with.

PETSc

Download PETSc 3.11.4

cd $STACK_SRC
curl -L -O http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.11.4.tar.gz
tar -xf petsc-3.11.4.tar.gz -C .

Now we configure, build, and install it

cd $STACK_SRC/petsc-3.11.4

./configure \
--prefix=$PACKAGES_DIR/petsc-3.11.4 \
--with-debugging=0 \
--with-ssl=0 \
--with-pic=1 \
--with-openmp=1 \
--with-mpi=1 \
--with-shared-libraries=1 \
--with-cxx-dialect=C++11 \
--with-fortran-bindings=0 \
--with-sowing=0 \
--download-hypre=1 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-ptscotch=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-scalapack=1 \
--download-mumps=1 \
--download-slepc=1 \
PETSC_DIR=`pwd` PETSC_ARCH=linux-opt

Once configure is done, we build PETSc

make PETSC_DIR=$STACK_SRC/petsc-3.11.4 PETSC_ARCH=linux-opt all

Everything good so far? PETSc should be asking to run more make commands

make PETSC_DIR=$STACK_SRC/petsc-3.11.4 PETSC_ARCH=linux-opt install

And now after the install, we can run some built-in tests

make PETSC_DIR=$PACKAGES_DIR/petsc-3.11.4 PETSC_ARCH="" test

Running the tests should produce some output like the following:

[moose@centos-8 petsc-3.11.4]$ make PETSC_DIR=$PACKAGES_DIR/petsc-3.11.4 PETSC_ARCH="" test
Running test examples to verify correct installation
Using PETSC_DIR=/opt/moose/petsc-3.11.4 and PETSC_ARCH=
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
=========================================

Create MOOSE Profile

Use an editor to add the following contents to $HOME/.moose_profile

PACKAGES_DIR=$HOME/moose-compilers

export CC=mpicc
export CXX=mpicxx
export F90=mpif90
export F77=mpif77
export FC=mpif90

export PETSC_DIR=$PACKAGES_DIR/petsc-3.11.4

Source the MOOSE Profile


source $HOME/.moose_profile

By sourcing the above file, you are now ready to begin MOOSE-based development.

note:Remember to source the profile!

You will need to perform the above (source $HOME/.moose_profile) for every new terminal session for which you perform work with MOOSE. If you want this to be automatic, add the above to your ~/.bash_profile (or ~/.bashrc or, which ever profile you use on your system)

Cleanup

Whith everything finished, it is now safe to remove the temporary directory containing the source tree:


if [ -d "$STACK_SRC" ]; then rm -rf "$STACK_SRC"; fi

Obtaining and Building MOOSE

Cloning MOOSE

MOOSE is hosted on GitHub and should be cloned directly from there using git. We recommend creating a directory named projects to put all of your MOOSE related work.

To install MOOSE run the following commands in a terminal.


mkdir ~/projects
cd ~/projects
git clone https://github.com/idaholab/moose.git
cd moose
git checkout master
note

The master branch of MOOSE is the stable branch that will only be updated after all tests are passing. This protects you from the day-to-day changes in the MOOSE repository.

Compile libMesh

MOOSE directly relies on the libMesh finite-element framework. Because of this strong tie MOOSE contains a particular version of libMesh that we have vetted for our users. To pull down and compile this version of libMesh you simply need to run a script in MOOSE:


cd ~/projects/moose
./scripts/update_and_rebuild_libmesh.sh
warning

Do not use sudo when running update_and_rebuild_libmesh.sh.

Compile and Test MOOSE


cd ~/projects/moose/test
make -j 4
./run_tests -j 4

If the installation was successful you should see most of the tests passing (some tests will be skipped depending on your system environment).

Head back over to the Getting Started page to continue your tour of MOOSE.