.. _containers:

==========
Containers
==========

.. contents:: In this section:
   :local:
   :depth: 2

Containers are a way to package software with all of its dependencies, creating isolated and reproducible environments. At GridUnesp, we use `Apptainer <https://apptainer.org/docs/user/main/>`_ (formerly Singularity) to run containers.

Why Use Containers?
===================

1. **Reproducibility:** The exact same environment can be used by different users
2. **Isolation:** Conflicting dependencies do not interfere with each other
3. **Portability:** The same container works on different systems
4. **Convenience:** Distribute your environment with a single file
5. **Security:** Apptainer containers do not require root privileges

Apptainer vs Docker
===================

Apptainer is similar to Docker, but designed for HPC:

=============== ================== ================================
Feature         Docker             Apptainer
--------------- ------------------ --------------------------------
Permissions     Requires root      Normal user
HPC Integration Limited            Native (MPI, GPUs)
Images          Layers             Single file (.sif)
Security        Less secure on HPC Designed for shared environments
=============== ================== ================================

Basic Concepts
==============

- **Image:** File containing the container's filesystem (``.sif``)
- **Definition:** Text file with instructions to build the image (``.def``)
- **Sandbox:** Directory for interactive development (``apptainer shell image.sif``)
- **Registry:** Image repository (Docker Hub, etc.)

Obtaining Images
================

Method 1: Pull from Docker Hub
------------------------------

Job to download and build the container (a file with the ".sif" suffix):

.. code-block:: bash
   :caption: pull_container.sh

   #!/bin/bash
   #SBATCH -J pull_container
   #SBATCH -t 01:00:00
   #SBATCH -n 1

   export INPUT=""
   export OUTPUT="ubuntu.sif"

   job-nanny apptainer pull ubuntu.sif docker://ubuntu:20.04

In this case, the ``ubuntu.sif`` container is built, which is an image of **Ubuntu 20.04**.


Method 2: Build from a definition file
--------------------------------------

Definition file (``ubuntu.def``):

.. code-block:: text
   :caption: ubuntu.def

   Bootstrap: docker
   From: ubuntu:20.04

   %post
       apt-get update
       apt-get install -y python3 python3-pip
       pip3 install numpy pandas matplotlib

   %environment
       export LC_ALL=C

   %runscript
       echo "Container Ubuntu com Python e pacotes científicos"
       python3 "$@"

Build script (``ubuntu.sif``):

.. code-block:: bash
   :caption: build_container.sh

   #!/bin/bash
   #SBATCH -J build_container
   #SBATCH -t 02:00:00
   #SBATCH -n 1
   #SBATCH --mem=4G

   export INPUT="ubuntu.def"
   export OUTPUT="ubuntu.sif"

   job-nanny apptainer build ubuntu.sif ubuntu.def

Running Containers
==================

Execution modes:

1. **apptainer run:** Runs the default command defined in the container
2. **apptainer exec:** Runs a specific command
3. **apptainer shell:** Opens an interactive shell

**Basic example:**

.. code-block:: bash
   :caption: run_container.sh

   #!/bin/bash
   #SBATCH -J run_container
   #SBATCH -t 01:00:00
   #SBATCH -n 1

   export INPUT="ubuntu.sif script.py dados/"
   export OUTPUT="resultados/"

   job-nanny apptainer exec ubuntu.sif python3 script.py

**Example with an interactive shell (for testing):**

.. code-block:: bash

   apptainer shell ubuntu.sif

   # Inside the container
   ls -la
   python3
   exit

Sharing Files
=============

By default, Apptainer automatically mounts:

- The user's ``$HOME``
- The current working directory
- The system's temporary directories

**Explicit mounting:**

.. code-block:: bash

   # Mount additional directories
   apptainer exec --bind /path/on/host:/path/in/container ubuntu.sif command

**With job-nanny:**

.. code-block:: bash
   :caption: container_com_dados.sh

   #!/bin/bash
   #SBATCH -J container_dados
   #SBATCH -t 02:00:00
   #SBATCH -n 1

   export INPUT="ubuntu.sif dados/ script.py"
   export OUTPUT="resultados/"

   # job-nanny copies INPUT to the work area
   # Inside the container, the files will be in the current directory
   job-nanny apptainer exec ubuntu.sif python3 script.py

Containers with GPU
===================

To use GPUs, add the ``--nv`` flag:

.. code-block:: bash
   :caption: container_gpu.def

   Bootstrap: docker
   From: nvidia/cuda:11.8.0-runtime-ubuntu20.04

   %post
       apt-get update
       apt-get install -y python3 python3-pip
       pip3 install torch torchvision

   %runscript
       python3 "$@"

Build:

.. code-block:: bash
   :caption: job_gpu_container.sh

   # Submit job
   #!/bin/bash
   #SBATCH -J gpu_container
   #SBATCH --partition=gpu
   #SBATCH --gres=gpu:1
   #SBATCH -t 02:00:00
   #SBATCH -n 1

   export INPUT="container_gpu.def treinamento.py"
   export OUTPUT="modelo.pth"

   module load cuda/11.8  # Optional, Apptainer uses the container's CUDA

   job-nanny apptainer build --nv cuda_container.sif container_gpu.def
   job-nanny apptainer exec --nv cuda_container.sif python3 treinamento.py

Execution:

.. code-block:: bash

   sbatch job_gpu_container.sh


Containers with MPI
===================

Apptainer can run MPI applications across multiple nodes.

**Definition file with MPI:**

.. code-block:: text
   :caption: mpi_container.def

   Bootstrap: docker
   From: ubuntu:22.04

   %environment
       export OMPI_DIR=/opt/ompi
       export PATH=$OMPI_DIR/bin:$PATH
       export LD_LIBRARY_PATH=$OMPI_DIR/lib:$LD_LIBRARY_PATH

   %post
       apt-get update
       apt-get install -y wget build-essential

       # Install OpenMPI
       export OMPI_DIR=/opt/ompi
       export OMPI_VERSION=4.1.5
       mkdir -p /tmp/ompi
       cd /tmp/ompi
       wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-$OMPI_VERSION.tar.bz2
       tar -xjf openmpi-$OMPI_VERSION.tar.bz2
       cd openmpi-$OMPI_VERSION
       ./configure --prefix=$OMPI_DIR
       make -j4 install

       # Install compilers and libraries
       apt-get install -y gcc g++ gfortran

   %files
       mpitest.c /opt/mpitest.c

**Example MPI code (mpitest.c):**

.. code-block:: c

   #include <mpi.h>
   #include <stdio.h>
   #include <unistd.h>

   int main(int argc, char** argv) {
       MPI_Init(&argc, &argv);

       int world_rank, world_size;
       char hostname[256];

       MPI_Comm_rank(MPI_COMM_WORLD, &world_rank);
       MPI_Comm_size(MPI_COMM_WORLD, &world_size);
       gethostname(hostname, sizeof(hostname));

       printf("Processo %d de %d em %s\n", world_rank, world_size, hostname);

       MPI_Finalize();
       return 0;
   }

**Scripts for building, compiling and running:**

.. code-block:: bash
   :caption: build_mpi_container.sh

   #!/bin/bash
   #SBATCH -J build_mpi
   #SBATCH -t 02:00:00
   #SBATCH -n 1

   export INPUT="mpi_container.def"
   export OUTPUT="mpi_container.sif"

   job-nanny apptainer build mpi_container.sif mpi_container.def

.. code-block:: bash
   :caption: compile_in_container.sh

   #!/bin/bash
   #SBATCH -J compile_mpi
   #SBATCH -t 00:30:00
   #SBATCH -n 1

   export INPUT="mpi_container.sif mpitest.c"
   export OUTPUT="mpitest"

   job-nanny apptainer exec mpi_container.sif mpicc -o mpitest /opt/mpitest.c

.. code-block:: bash
   :caption: run_mpi_container.sh

   #!/bin/bash
   #SBATCH -J run_mpi
   #SBATCH -N 4
   #SBATCH --ntasks-per-node=28
   #SBATCH -t 01:00:00
   #SBATCH --mem-per-cpu=2G

   export INPUT="mpi_container.sif mpitest"
   export OUTPUT="mpi_output/"

   module load openmpi/4.1.5  # System MPI for orchestration

   # --sharens avoids namespace conflicts
   job-nanny mpirun -n 112 apptainer exec --sharens mpi_container.sif ./mpitest

This job will run 112 processes, 28 of them on each node. The *OpenMPI* used is the one already installed in the container (mpi_container.sif), and the executable (mpitest) is processed inside that container. The MPI installed in the container runs in conjunction with the system one, loaded via module (``module load openmpi/4.1.5``). Note that the *OpenMPI* version installed in the container, for the example above, is **4.1.5**, which is the same version as the module. The versions of both *OpenMPI* installations must be compatible.

.. attention::
   According to the `Using –sharens mode <https://apptainer.org/docs/user/main/mpi.html#using-sharens-mode>`_ section of the Apptainer page, in order to ensure there are no *namespace*-related conflicts during parallelization, it is recommended to include the ``--sharens`` parameter right after the ``exec`` command. Like this: ``apptainer exec --sharens``.


Containers for Specific Applications
====================================

Example: GROMACS in a container
-------------------------------

.. code-block:: text
   :caption: gromacs.def

   Bootstrap: docker
   From: ubuntu:22.04

   %post
       apt-get update
       apt-get install -y wget cmake build-essential

       # Install GROMACS
       wget ftp://ftp.gromacs.org/pub/gromacs/gromacs-2023.tar.gz
       tar -xzf gromacs-2023.tar.gz
       cd gromacs-2023
       mkdir build
       cd build
       cmake .. -DGMX_BUILD_OWN_FFTW=ON -DCMAKE_INSTALL_PREFIX=/usr/local/gromacs
       make -j4
       make install

   %environment
       export PATH=/usr/local/gromacs/bin:$PATH
       export LD_LIBRARY_PATH=/usr/local/gromacs/lib:$LD_LIBRARY_PATH

.. code-block:: bash
   :caption: build_gromacs.sh

   #!/bin/bash
   #SBATCH -J build_gromacs
   #SBATCH -t 04:00:00
   #SBATCH -n 1

   export INPUT="gromacs.def"
   export OUTPUT="gromacs.sif"

   job-nanny apptainer build gromacs.sif gromacs.def

.. code-block:: bash
   :caption: run_gromacs.sh

   #!/bin/bash
   #SBATCH -J gromacs_container
   #SBATCH -N 2
   #SBATCH --ntasks-per-node=28
   #SBATCH -t 24:00:00

   export INPUT="gromacs.sif topol.tpr"
   export OUTPUT="resultado_gromacs/"

   module load openmpi/4.1.5

   job-nanny mpirun -n 56 apptainer exec gromacs.sif mdrun_mpi -deffnm resultado

Example: Python with specific libraries
---------------------------------------

.. code-block:: text
   :caption: python_sci.def

   Bootstrap: docker
   From: python:3.9-slim

   %post
       pip install --upgrade pip
       pip install numpy scipy pandas matplotlib scikit-learn jupyter

   %runscript
       python "$@"

.. code-block:: bash
   :caption: build_python_sci.sh

   #!/bin/bash
   #SBATCH -J build_python_sci
   #SBATCH -t 02:00:00
   #SBATCH -n 1

   export INPUT="python_sci.def"
   export OUTPUT="python_sci.sif"

   job-nanny apptainer build python_sci.sif python_sci.def

.. code-block:: bash
   :caption: run_python.sh

   #!/bin/bash
   #SBATCH -J python_container
   #SBATCH -n 1
   #SBATCH -t 02:00:00

   export INPUT="python_sci.sif analise.py dados.csv"
   export OUTPUT="resultados_analise/"

   job-nanny apptainer exec python_sci.sif python analise.py

Best Practices with Containers
==============================

1. **Lightweight images**

   - Use minimal base images (`slim <https://www.cloudbees.com/blog/alpine-based-docker-images-make-difference-real-world-apps>`_, `alpine <https://hub.docker.com/_/alpine>`_)
   - Remove unnecessary caches in %post

     .. code-block:: text

        %post

             apt-get update && apt-get install -y \
             git \
             python3 \
             && rm -rf /var/lib/apt/lists/*

2. **Versioning**

   .. code-block:: text

      Bootstrap: docker
      From: ubuntu:20.04  # Fixed version, not "latest"

3. **Reproducibility**

   - Include exact package versions
   - Document the build process

4. **Security**

   - Do not put passwords or keys in the image
   - Use environment variables for sensitive settings

5. **Organization**

   .. code-block:: text

      meus_containers/
      ├── ubuntu-python/
      │   ├── ubuntu-python.def
      │   ├── build.slurm
      │   └── ubuntu-python.sif
      ├── gromacs/
      │   ├── gromacs.def
      │   └── build.slurm
      └── mpi-base/
          ├── mpi.def
          └── README.md

6. **Cleanup**

   - Remove unnecessary caches in ``/home/$USER/.apptainer/``

     .. code-block:: text

        apptainer cache clean

Troubleshooting
===============

**Error: "no space left on device" during build**

- Use the ``job-nanny`` option ``LARGE_FILES="true"``.

  .. code-block:: bash

     #!/bin/bash
     #SBATCH -J build_ubuntu
     #SBATCH -t 02:00:00
     #SBATCH -n 1

     export INPUT="ubuntu.def"
     export OUTPUT="ubuntu.sif"

     export LARGE_FILES="true"

     job-nanny apptainer build ubuntu.sif ubuntu.def

**Error: "permission denied"**

- Check the .sif file permissions
- Make sure the container does not require root

**Error: "GPU not available" inside the container**

- Use the ``--nv`` flag
- Check whether CUDA is installed in the container

**Error: "MPI version mismatch"**

- Use the ``--sharens`` flag
- Compile MPI in the container with the same version as the system (or a lower version)

.. seealso::

   - :ref:`using_job_nanny`
   - :ref:`installing_applications` - Installation alternatives
   - :ref:`gpu_usage`
   - :ref:`distributed_memory` - MPI
   - `Apptainer Documentation <https://apptainer.org/docs/user/main/>`_