Multi-User Cluster Instructions

This document will aid an HPC Administrator on building an environment which multiple end-users will use when compiling and running MOOSE based applications. If you do not have administrative rights, you will not be able to complete these instructions! Please forward these instructions to your HPC Administrator.

Because this document assumes the reader is an administrator, most of the content herein are suggestions, and not 'copy, paste, enter' instructions.


  • Some sort of environmental module management software. Such as Environment Modules.

  • A working MPI wrapper (MPICH/OpenMPI/MVAPICH) which wraps to a C++11 compliant compiler.

  • Knowledge on Access Control Lists (ACLs) or other means to safeguard a directory from further tampering by others.


Please use a single solitary terminal session throughout and to the completion of these instructions.

Environment Setup

Begin by creating an area for which to build:

export STACK_SRC=`mktemp -d /tmp/stack_src_temp.XXXXXX`

Set your umask

Some systems have a secure umask set. We need to adjust our umask so that when you write a file (make install), it is readable by everyone:

umask 0022

Choose a base path

Export a base path variable which will be the home location for the compiler stack. All files related to libraries necessary to build MOOSE, will be stored in this location (choose carefully, as this location should be accessible from all nodes on your cluster):

export PACKAGES_DIR=/opt/moose

Create and chown $PACKGES_DIR

History teaches us, that implicitly trusting scripts we download off the internet with root access, is a very bad idea. So let us create and chown the $PACKAGES_DIR directory before we install anything. That way, the things we do install can be done so without invoking sudo:

sudo mkdir -p $PACKAGES_DIR
sudo chown -R <your user id> $PACKAGES_DIR

Verify that your umask settings are indeed set to 0022 before continuing:

$> umask

Set Up Modules

Even if you are not using Modules, the following provides information on what environment variables are needed for MOOSE developement.

Create a MOOSE module:

mkdir -p $PACKAGES_DIR/modulefiles
vi $PACKAGES_DIR/modulefiles/moose-dev-gcc

Add the following content to that file:

## MOOSE module



prepend-path    PATH             /GCC and MPI /bin
prepend-path    LD_LIBRARY_PATH  /GCC and MPI /lib

setenv CC       mpicc
setenv CXX      mpicxx
setenv F90      mpif90
setenv F77      mpif77
setenv FC       mpif90

setenv          PETSC_DIR        $base_path/petsc/petsc-3.8.3

# Optional if miniconda is installed
prepend-path    PATH             $base_path/miniconda/bin

Replace INSERT PACKAGES_DIR HERE with whatever you had set for $PACKAGES_DIR (do not literally enter: $PACKAGES_DIR. As an example, if you left packages_dir as: /opt/moose, then that is what you would enter)

Replace GCC and MPI paths with any additional information needed to make GCC/MPI work on your cluster.

To make the module available in your terminal session, export the following:


The above command should be added to the list of other global profiles (perhaps in /etc/profiles.d). That way, the above is performed as the user logs into the machine.

With the modulefile in place and the MODULEPATH variable set, see if our module is available for loading:

module load moose-dev-gcc

Verify that this module loads properly by attempting to echo $PETSC_DIR:


While we are at it, verify that your MPI wrapper works by running a few commands (your results will vary, but they should return something):

which mpicc

mpicc -show
gcc -I/opt/moose/mpich-3.2/gcc-7.3.0/include -L/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,-rpath -Wl,/opt/moose/mpich-3.2/gcc-7.3.0/lib -Wl,--enable-new-dtags -lmpi

which gcc

Leave this module loaded for the remainder of the instructions (PETSc requirements).


Download PETSc 3.8.3

curl -L -O
tar -xf petsc-3.8.3.tar.gz -C .

Now we configure, build, and install it

cd $STACK_SRC/petsc-3.8.3

./configure \
--prefix=$PACKAGES_DIR/petsc-3.8.3 \
--download-hypre=1 \
--with-ssl=0 \
--with-debugging=no \
--with-pic=1 \
--with-shared-libraries=1 \
--with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-mumps=1 \
--download-scalapack=1 \
--CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 \
--CFLAGS='-fPIC -fopenmp' \
--CXXFLAGS='-fPIC -fopenmp' \
--FFLAGS='-fPIC -fopenmp' \
--FCFLAGS='-fPIC -fopenmp' \
--F90FLAGS='-fPIC -fopenmp' \
--F77FLAGS='-fPIC -fopenmp' \

Once configure is done, we build PETSc

make PETSC_DIR=$STACK_SRC/petsc-3.8.3 PETSC_ARCH=arch-linux2-c-opt all

Everything good so far? PETSc should be asking to run more make commands

make PETSC_DIR=$STACK_SRC/petsc-3.8.3 PETSC_ARCH=arch-linux2-c-opt install

And now after the install, we can run some built-in tests

make PETSC_DIR=$PACKAGES_DIR/petsc-3.8.3 PETSC_ARCH="" test

Running the tests should produce some output like the following:

[moose@centos-7 petsc-3.8.3]$ make PETSC_DIR=$PACKAGES_DIR/petsc-3.8.3 PETSC_ARCH="" test
Running test examples to verify correct installation
Using PETSC_DIR=/opt/moose/petsc-3.8.3 and PETSC_ARCH=
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples


Peacock (an optional MOOSE GUI frontend) uses many libraries. The easiest way to obtain these libraries, is to install miniconda, along with several miniconda/pip packages.

curl -L -O
sh -b -p $PACKAGES_DIR/miniconda

PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda config --set ssl_verify false
PATH=$PACKAGES_DIR/miniconda/bin:$PATH conda install -c idaholab python=2.7 coverage \
reportlab \
mako \
numpy \
scipy \
scikit-learn \
h5py \
hdf5 \
scikit-image \
requests \
vtk=7.1.0 \
pyyaml \
matplotlib \
pip \
lxml \
pyflakes \
pandas \
conda-build \
mock \
yaml \
pyqt \
swig --yes

Peacock (as well as the TestHarness sytem in MOOSE), does not work with Python3. Please chose Miniconda2 for Python 2.7 instead.

Next, we need to use pip to install additional libraries not supplied by conda:

PATH=$PACKAGES_DIR/miniconda/bin:$PATH pip install --no-cache-dir pybtex livereload==2.5.1 daemonlite pylint==1.6.5 lxml pylatexenc anytree

Clean Up and Chown

Clean all the temporary stuff and change the ownership to root, so no further writes are possible:

rm -rf $STACK_SRC
sudo chown -R root:root $PACKAGE_DIR

This concludes setting up the environment for MOOSE-based development. However you decide to instruct your users on enabling the above environment, each user will need to perform the instructions provided by the following link: Obtaining and Building MOOSE.