Libtorch (Pytorch C++ API)

The way one enables LibTorch Paszke et al. (2019) capabilities in MOOSE depends on the operating system (Linux or Mac) and if we use HPC or just a local workstation.

note:Compatibility

Before trying to install MOOSE with libtorch make sure to check out the compatibility matrix which will tell you if the required packages are compatible with the ones required by MOOSE. Furthermore, we do not support libtorch versions below v1.4.

Mac workstations

For Mac workstations, the user has two distinct paths for enabling libtorch within MOOSE. Both paths rely on conda for installing the moose environment, therefore we recommend the users follow Conda MOOSE Environment instructions before starting this process.

note:GPU Support on Macs

For ARM Mac workstations, both installation procedures will give access to the Metal Performance Shader (MPS) capabilities (GPU acceleration).

Using packages distributed through conda

Create and activate a new MOOSE environment with pytorch included:
```
conda create -n moose-torch moose-dev pytorch=2.1 -c pytorch
conda activate moose-torch
```
This will provide the headers and libraries needed to use libtorch.
Configure MOOSE to use libtorch. The user needs to link to the conda-based libtorch libraries using the following command within the root directory of moose:
```
./configure --with-libtorch=${CONDA_PREFIX}/lib/python3.11/site-packages/torch
```
note:How to get the pytorch directory
The python version can be different depending on the distribution, so make sure you double-check if the directory you point to actually exists! An easy way to find if the library exists within the conda package is running the following command in the terminal:
find ${CONDA_PREFIX} -type d -name torch
Alternatively, you can use python to get the same directory:
python -c "import torch; print(torch.__path__[0])"
Once moose has been configured to work with libtorch, we need to recompile MOOSE. For testing purposes, we can do the following:
```
cd test
make -j 8
./run_tests --re=libtorch
```
If you see tests passing with green colors, the installation was successful.

Using pre-compiled packages from the official website

Create and activate a MOOSE environment without pytorch:
```
conda create -n moose moose-dev
conda activate moose
```
Pull the precompiled libraries using the setup script provided in MOOSE. From the root directory of moose this is done by executing the following command:
```
./scripts/setup_libtorch.sh --version=2.1
```
note:Version limitations for ARM Workstations
The official distribution system does not include packages for ARM architectures up until version 2.2. If the user want to use this path on ARM machines, the version parameter to the script needs to be altered.
Configure MOOSE to use libtorch within the root directory of moose:
```
./configure --with-libtorch
```
In this case, the path is not added explicitly considering that the libtorch packages will be pulled to the default location.
Once moose has been configured to work with libtorch, we need to recompile MOOSE. For testing purposes, we can do the following:
```
cd test
make -j 8
./run_tests --re=libtorch
```
If you see tests passing with green colors, the installation was successful.

Linux Workstations

For linux, due to the official conda distribution of pytorch is using pre-CXX11 ABI, while the conda compiler stack of MOOSE relies on this ABI, we don't support conda-based installations yet.

note:Main limitation on Linux machines

It is important to emphasize that linking MOOSE with libtorch on Linux machines is not supported if the compiler stack has been built using a libc version below 2.29 (for libtorch v 2.1+) or 2.27 (for libtorch v 1.8-2.1) or 2.23 (for libtorch v1.4-1.8). To check your currently used libc version on Linux machine, use the following command:

ldd --version

This can be a problem when using the moose conda environment with new versions of libtorch.

Using pre-compiled packages from the official website

Create and activate a MOOSE environment without pytorch:
```
conda create -n moose moose-dev
conda activate moose
```
Pull the precompiled libraries using the setup script provided in MOOSE. From the root directory of moose this is done by executing the following command:
```
./scripts/setup_libtorch.sh --version=2.1
```
Configure MOOSE to use libtorch within the root directory of moose:
```
./configure --with-libtorch
```
Once moose has been configured to work with libtorch, we need to recompile MOOSE. For testing purposes, we can do the following:
```
cd test
make -j 8
./run_tests --re=libtorch
```
If you see tests passing with green colors, the installation was successful.

Manually installing packages

When encountering GLIBC-related compatibility issues on Linux machines the user has two options:

Rebuilding Petsc and libMesh manually using compatible compilers (most newer systems like Ubuntu come with compatible compilers).
Building libtorch from source. The user can find instructions on how to install libtorch from source on the official website.

note:GPU Support on Linux Workstations

For linux workstations, we recommend manually building the dependencies of MOOSE using suitable system compilers. At the time these instruction are written, only cuda-based acceleration is tested. The following packages need to be also installed to enable this feature:

A sutiable Nvidia driver
Cuda toolkit - only strictly required if building libtorch from source.

The supported versions can be determined using the compatibility matrix. Once the dependencies of MOOSE are installed, we can use the setup script to fetch the correct libtorch package from the official ditribution:

./scripts/setup_libtorch.sh --version=2.1 --libtorch-distribution=cuda

The configuration and build parts of the process are the same as discussed before.

HPC systems

On non-INL HPC systems, one can follow the manual installation process discussed above. On INL machines, containers are provided with readily compiled dependencies, including libtorch. For more information on containers, see the instuctions. In this case, the moose-dev module already contains libtorch.

References

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: an imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d\textquotesingle Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019. URL: http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.

@incollection{paszke2019pytorch,
    author = "Paszke, Adam and Gross, Sam and Massa, Francisco and Lerer, Adam and Bradbury, James and Chanan, Gregory and Killeen, Trevor and Lin, Zeming and Gimelshein, Natalia and Antiga, Luca and Desmaison, Alban and Kopf, Andreas and Yang, Edward and DeVito, Zachary and Raison, Martin and Tejani, Alykhan and Chilamkurthy, Sasank and Steiner, Benoit and Fang, Lu and Bai, Junjie and Chintala, Soumith",
    editor = "Wallach, H. and Larochelle, H. and Beygelzimer, A. and d\textquotesingle Alch\'{e}-Buc, F. and Fox, E. and Garnett, R.",
    title = "PyTorch: An Imperative Style, High-Performance Deep Learning Library",
    booktitle = "Advances in Neural Information Processing Systems 32",
    pages = "8024--8035",
    year = "2019",
    publisher = "Curran Associates, Inc.",
    url = "http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf"
}

Mac workstations
Linux Workstations
HPC systems
References