HPC Cluster

The following instructions aims at setting up a baseline single-user environment for building MOOSE based applications in a job scheduling capable environment.

If instead, you are here interested in allowing MOOSE-based development to be made avaialable to multiple users, please see our Multi-User setup instructions (requires administrative rights). Normal users: please ingore the previous sentence, and continue on...

Pre-Reqs

What ever compiler you choose to use on your cluster (GCC/Clang, MPICH/OpenMPI), the minimum requirement, is that it must be C++11 compatible. If you are unsure, please consult with your system admins for your cluster on which compiler to use (and how to use it).

  • CMake. A modern version of CMake (>2.8) is required to build some of the meta packages we need to include in PETSc.

  • Python 2.7.x Development libraries.

Your cluster will most likely have these two requirements available via some form of environment management software. If you are unfamiliar with how to manage your environment, please consult with your cluster administrators.

PREFIX Setup


export STACK_SRC=`mktemp -d /tmp/stack_temp.XXXXXX`
export PACKAGES_DIR=$HOME/moose-compilers
note

  • The $PACKAGES_DIR directory must reside in a location where all the compute nodes can access.

  • We need the above two environment variables to exist throughout these instructions. Please use the one and only one, terminal you executed those two commands with.

PETSc

Download PETSc 3.8.3

cd $STACK_SRC
curl -L -O http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-__PETSC_DEFAULT__.tar.gz
tar -xf petsc-__PETSC_DEFAULT__.tar.gz -C .

Now we configure, build, and install it

cd $STACK_SRC/petsc-__PETSC_DEFAULT__

./configure \
--prefix=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__ \
--download-hypre=1 \
--with-ssl=0 \
--with-debugging=no \
--with-pic=1 \
--with-shared-libraries=1 \
--with-cc=mpicc \
--with-cxx=mpicxx \
--with-fc=mpif90 \
--download-fblaslapack=1 \
--download-metis=1 \
--download-parmetis=1 \
--download-superlu_dist=1 \
--download-mumps=1 \
--download-scalapack=1 \
--CC=mpicc --CXX=mpicxx --FC=mpif90 --F77=mpif77 --F90=mpif90 \
--CFLAGS='-fPIC -fopenmp' \
--CXXFLAGS='-fPIC -fopenmp' \
--FFLAGS='-fPIC -fopenmp' \
--FCFLAGS='-fPIC -fopenmp' \
--F90FLAGS='-fPIC -fopenmp' \
--F77FLAGS='-fPIC -fopenmp' \
PETSC_DIR=`pwd`

Once configure is done, we build PETSc

make PETSC_DIR=$STACK_SRC/petsc-__PETSC_DEFAULT__ PETSC_ARCH=arch-linux2-c-opt all

Everything good so far? PETSc should be asking to run more make commands

make PETSC_DIR=$STACK_SRC/petsc-__PETSC_DEFAULT__ PETSC_ARCH=arch-linux2-c-opt install

And now after the install, we can run some built-in tests

make PETSC_DIR=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__ PETSC_ARCH="" test

Running the tests should produce some output like the following:

[moose@centos-7 petsc-__PETSC_DEFAULT__]$ make PETSC_DIR=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__ PETSC_ARCH="" test
Running test examples to verify correct installation
Using PETSC_DIR=/opt/moose/petsc-__PETSC_DEFAULT__ and PETSC_ARCH=
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
Fortran example src/snes/examples/tutorials/ex5f run successfully with 1 MPI process
Completed test examples
=========================================

Create MOOSE Profile

Use an editor to add the following contents to $HOME/.moose_profile

PACKAGES_DIR=$HOME/moose-compilers

export CC=mpicc
export CXX=mpicxx
export F90=mpif90
export F77=mpif77
export FC=mpif90

export PETSC_DIR=$PACKAGES_DIR/petsc-__PETSC_DEFAULT__

Source the MOOSE Profile


source $HOME/.moose_profile

By sourcing the above file, you are now ready to begin MOOSE-based development.

note:Remember to source the profile!

You will need to perform the above (source $HOME/.moose_profile) for every new terminal session for which you perform work with MOOSE. If you want this to be automatic, add the above to your ~/.bash_profile (or ~/.bashrc or, which ever profile you use on your system)

Cleanup

Whith everything finished, it is now safe to remove the temporary directory containing the source tree:


if [ -d "$STACK_SRC" ]; then rm -rf "$STACK_SRC"; fi

Head back over to the Getting Started page to continue your tour of MOOSE.