Containers

In this section:

Why Use Containers?
Apptainer vs Docker
Basic Concepts
Obtaining Images
- Method 1: Pull from Docker Hub
- Method 2: Build from a definition file
Running Containers
Sharing Files
Containers with GPU
Containers with MPI
Containers for Specific Applications
- Example: GROMACS in a container
- Example: Python with specific libraries
Best Practices with Containers
Troubleshooting

Containers are a way to package software with all of its dependencies, creating isolated and reproducible environments. At GridUnesp, we use Apptainer (formerly Singularity) to run containers.

Why Use Containers?

Reproducibility: The exact same environment can be used by different users
Isolation: Conflicting dependencies do not interfere with each other
Portability: The same container works on different systems
Convenience: Distribute your environment with a single file
Security: Apptainer containers do not require root privileges

Apptainer vs Docker 

Apptainer is similar to Docker, but designed for HPC:

Feature	Docker	Apptainer
Permissions	Requires root	Normal user
HPC Integration	Limited	Native (MPI, GPUs)
Images	Layers	Single file (.sif)
Security	Less secure on HPC	Designed for shared environments

Basic Concepts 

Image: File containing the container’s filesystem (.sif)
Definition: Text file with instructions to build the image (.def)
Sandbox: Directory for interactive development (apptainer shell image.sif)
Registry: Image repository (Docker Hub, etc.)

Obtaining Images 

Method 1: Pull from Docker Hub 

Job to download and build the container (a file with the “.sif” suffix):

pull_container.sh

#!/bin/bash
#SBATCH -J pull_container
#SBATCH -t 01:00:00
#SBATCH -n 1

export INPUT=""
export OUTPUT="ubuntu.sif"

job-nanny apptainer pull ubuntu.sif docker://ubuntu:20.04

In this case, the ubuntu.sif container is built, which is an image of Ubuntu 20.04.

Method 2: Build from a definition file 

Definition file (ubuntu.def):

ubuntu.def

Bootstrap: docker
From: ubuntu:20.04

%post
    apt-get update
    apt-get install -y python3 python3-pip
    pip3 install numpy pandas matplotlib

%environment
    export LC_ALL=C

%runscript
    echo "Container Ubuntu com Python e pacotes científicos"
    python3 "$@"

Build script (ubuntu.sif):

build_container.sh

#!/bin/bash
#SBATCH -J build_container
#SBATCH -t 02:00:00
#SBATCH -n 1
#SBATCH --mem=4G

export INPUT="ubuntu.def"
export OUTPUT="ubuntu.sif"

job-nanny apptainer build ubuntu.sif ubuntu.def

Running Containers 

Execution modes:

apptainer run: Runs the default command defined in the container
apptainer exec: Runs a specific command
apptainer shell: Opens an interactive shell

Basic example:

run_container.sh

#!/bin/bash
#SBATCH -J run_container
#SBATCH -t 01:00:00
#SBATCH -n 1

export INPUT="ubuntu.sif script.py dados/"
export OUTPUT="resultados/"

job-nanny apptainer exec ubuntu.sif python3 script.py

Example with an interactive shell (for testing):

apptainer shell ubuntu.sif

# Inside the container
ls -la
python3
exit

Containers with GPU 

To use GPUs, add the --nv flag:

container_gpu.def

Bootstrap: docker
From: nvidia/cuda:11.8.0-runtime-ubuntu20.04

%post
    apt-get update
    apt-get install -y python3 python3-pip
    pip3 install torch torchvision

%runscript
    python3 "$@"

Build:

job_gpu_container.sh

# Submit job
#!/bin/bash
#SBATCH -J gpu_container
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH -t 02:00:00
#SBATCH -n 1

export INPUT="container_gpu.def treinamento.py"
export OUTPUT="modelo.pth"

module load cuda/11.8  # Optional, Apptainer uses the container's CUDA

job-nanny apptainer build --nv cuda_container.sif container_gpu.def
job-nanny apptainer exec --nv cuda_container.sif python3 treinamento.py

Execution:

sbatch job_gpu_container.sh

Containers with MPI 

Apptainer can run MPI applications across multiple nodes.

Definition file with MPI:

mpi_container.def

Bootstrap: docker
From: ubuntu:22.04

%environment
    export OMPI_DIR=/opt/ompi
    export PATH=$OMPI_DIR/bin:$PATH
    export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH

%post
    apt-get update
    apt-get install -y wget build-essential

    # Install OpenMPI
    export OMPI_DIR=/opt/ompi
    export OMPI_VERSION=4.1.5
    mkdir -p /tmp/ompi
    cd /tmp/ompi
    wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2
    tar -xjf openmpi-$OMPI_VERSION.tar.bz2
    cd openmpi-$OMPI_VERSION
    ./configure --prefix=$OMPI_DIR
    make -j4 install

    # Install compilers and libraries
    apt-get install -y gcc g++ gfortran

%files
    mpitest.c /opt/mpitest.c

Example MPI code (mpitest.c):

#include <mpi.h>
#include <stdio.h>
#include <unistd.h>

int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);

    int world_rank, world_size;
    char hostname[256];

    MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
    MPI_Comm_size(MPI_COMM_WORLD, &world_size);
    gethostname(hostname, sizeof(hostname));

    printf("Processo %d de %d em %s\n", world_rank, world_size, hostname);

    MPI_Finalize();
    return 0;
}

Scripts for building, compiling and running:

build_mpi_container.sh

#!/bin/bash
#SBATCH -J build_mpi
#SBATCH -t 02:00:00
#SBATCH -n 1

export INPUT="mpi_container.def"
export OUTPUT="mpi_container.sif"

job-nanny apptainer build mpi_container.sif mpi_container.def

compile_in_container.sh

#!/bin/bash
#SBATCH -J compile_mpi
#SBATCH -t 00:30:00
#SBATCH -n 1

export INPUT="mpi_container.sif mpitest.c"
export OUTPUT="mpitest"

job-nanny apptainer exec mpi_container.sif mpicc -o mpitest /opt/mpitest.c

run_mpi_container.sh

#!/bin/bash
#SBATCH -J run_mpi
#SBATCH -N 4
#SBATCH --ntasks-per-node=28
#SBATCH -t 01:00:00
#SBATCH --mem-per-cpu=2G

export INPUT="mpi_container.sif mpitest"
export OUTPUT="mpi_output/"

module load openmpi/4.1.5  # System MPI for orchestration

# --sharens avoids namespace conflicts
job-nanny mpirun -n 112 apptainer exec --sharens mpi_container.sif ./mpitest

This job will run 112 processes, 28 of them on each node. The OpenMPI used is the one already installed in the container (mpi_container.sif), and the executable (mpitest) is processed inside that container. The MPI installed in the container runs in conjunction with the system one, loaded via module (module load openmpi/4.1.5). Note that the OpenMPI version installed in the container, for the example above, is 4.1.5, which is the same version as the module. The versions of both OpenMPI installations must be compatible.

Attention

According to the Using –sharens mode section of the Apptainer page, in order to ensure there are no namespace-related conflicts during parallelization, it is recommended to include the --sharens parameter right after the exec command. Like this: apptainer exec --sharens.

Containers for Specific Applications 

Example: GROMACS in a container 

gromacs.def

Bootstrap: docker
From: ubuntu:22.04

%post
    apt-get update
    apt-get install -y wget cmake build-essential

    # Install GROMACS
    wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-2023.tar.gz
    tar -xzf gromacs-2023.tar.gz
    cd gromacs-2023
    mkdir build
    cd build
    cmake .. -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs
    make -j4
    make install

%environment
    export PATH=/usr/local/gromacs/bin:$PATH
    export LD_LIBRARY_PATH=/usr/local/gromacs/lib:$LD_LIBRARY_PATH

build_gromacs.sh

#!/bin/bash
#SBATCH -J build_gromacs
#SBATCH -t 04:00:00
#SBATCH -n 1

export INPUT="gromacs.def"
export OUTPUT="gromacs.sif"

job-nanny apptainer build gromacs.sif gromacs.def

run_gromacs.sh

#!/bin/bash
#SBATCH -J gromacs_container
#SBATCH -N 2
#SBATCH --ntasks-per-node=28
#SBATCH -t 24:00:00

export INPUT="gromacs.sif topol.tpr"
export OUTPUT="resultado_gromacs/"

module load openmpi/4.1.5

job-nanny mpirun -n 56 apptainer exec gromacs.sif mdrun_mpi -deffnm resultado

Example: Python with specific libraries 

python_sci.def

Bootstrap: docker
From: python:3.9-slim

%post
    pip install --upgrade pip
    pip install numpy scipy pandas matplotlib scikit-learn jupyter

%runscript
    python "$@"

build_python_sci.sh

#!/bin/bash
#SBATCH -J build_python_sci
#SBATCH -t 02:00:00
#SBATCH -n 1

export INPUT="python_sci.def"
export OUTPUT="python_sci.sif"

job-nanny apptainer build python_sci.sif python_sci.def

run_python.sh

#!/bin/bash
#SBATCH -J python_container
#SBATCH -n 1
#SBATCH -t 02:00:00

export INPUT="python_sci.sif analise.py dados.csv"
export OUTPUT="resultados_analise/"

job-nanny apptainer exec python_sci.sif python analise.py

Best Practices with Containers 

Lightweight images

Use minimal base images (slim, alpine)

Remove unnecessary caches in %post

%post

     apt-get update && apt-get install -y \
     git \
     python3 \
     && rm -rf /var/lib/apt/lists/*

Versioning

Bootstrap: docker
From: ubuntu:20.04  # Fixed version, not "latest"

Reproducibility
- Include exact package versions
- Document the build process
Security
- Do not put passwords or keys in the image
- Use environment variables for sensitive settings

Organization

meus_containers/
├── ubuntu-python/
│   ├── ubuntu-python.def
│   ├── build.slurm
│   └── ubuntu-python.sif
├── gromacs/
│   ├── gromacs.def
│   └── build.slurm
└── mpi-base/
    ├── mpi.def
    └── README.md

Cleanup
- Remove unnecessary caches in /home/$USER/.apptainer/
```
apptainer cache clean
```

Troubleshooting 

Error: “no space left on device” during build

Use the job-nanny option LARGE_FILES="true".

#!/bin/bash
#SBATCH -J build_ubuntu
#SBATCH -t 02:00:00
#SBATCH -n 1

export INPUT="ubuntu.def"
export OUTPUT="ubuntu.sif"

export LARGE_FILES="true"

job-nanny apptainer build ubuntu.sif ubuntu.def

Error: “permission denied”

Check the .sif file permissions
Make sure the container does not require root

Error: “GPU not available” inside the container

Use the --nv flag
Check whether CUDA is installed in the container

Error: “MPI version mismatch”

Use the --sharens flag
Compile MPI in the container with the same version as the system (or a lower version)