.. _job_array:

=========
Job Array
=========

.. contents:: In this section:
   :local:
   :depth: 2

A Job Array is a SLURM feature that lets you submit multiple similar jobs with a single script. It is ideal for **High Throughput Computing (HTC)**.

Concept
=======

A job array is a set of independent jobs, each with its own index. They all share the same submission script but can process different data based on the index.

**Analogy:** You hire a clothing company (the HPC cluster) and hand over a single pattern (the job script) and a list of parameters (color names or a range of numbers, such as 0 to 10). The company (Slurm) examines the pattern and assigns 8 seamstresses (CPUs) to work (process) simultaneously, each one taking a different parameter (color) from the list using ``$SLURM_ARRAY_TASK_ID``.

When to Use a Job Array
=======================

✅ **Ideal for:**

- Processing many independent files
- Parameter sweeps
- Monte Carlo simulations
- Embarrassingly parallel data analysis
- Bootstrapping and cross-validation

❌ **Not recommended for:**

- Jobs with simultaneous dependency on one another
- Communication between jobs
- Problems that require synchronization

Basic Syntax
============

.. code-block:: bash

   #!/bin/bash
   #SBATCH --array=1-100
   #SBATCH -n 1
   #SBATCH -t 01:00:00

   echo "My index is $SLURM_ARRAY_TASK_ID"

Array Formats
=============

1. **Simple range:** ``--array=1-100`` (indices 1 to 100)
2. **Specific list:** ``--array=1,5,10,20,50``
3. **With step:** ``--array=1-100:5`` (1,6,11,...,96)
4. **With simultaneous limit:** ``--array=1-1000%50`` (maximum of 50 jobs running at the same time)

Practical Examples
==================

Example 1: Processing Multiple Files
------------------------------------

Suppose you have 100 input files named ``data_1.dat``, ``data_2.dat``, ..., ``data_100.dat``.

.. code-block:: bash
   :caption: process_array.sh

   #!/bin/bash
   #SBATCH -J process_data
   #SBATCH --array=1-100
   #SBATCH -n 1
   #SBATCH -t 02:00:00
   #SBATCH --mem=4G
   #SBATCH --output=logs/array_%A_%a.out
   #SBATCH --error=logs/array_%A_%a.err

   # Job array variables
   # %A = Array Job ID
   # %a = Task ID (index)

   mkdir -p results/

   export INPUT="data_${SLURM_ARRAY_TASK_ID}.dat"
   export OUTPUT="results/result_${SLURM_ARRAY_TASK_ID}.dat"

   module load python/3.9
   job-nanny python process.py $INPUT -o $OUTPUT

Example 2: Parameter Sweep
--------------------------

To test different parameter combinations:

.. code-block:: bash
   :caption: sweep_array.sh

   #!/bin/bash
   #SBATCH -J sweep
   #SBATCH --array=1-27
   #SBATCH -n 1
   #SBATCH -t 04:00:00

   # Define parameter combinations
   temperature=(300 310 320 330 340 350 360 370 380)
   pressure=(1.0 1.5 2.0)
   catalyst=("Fe" "Ni" "Co")

   # Compute indices
   i_temp=$(( ($SLURM_ARRAY_TASK_ID - 1) / 3 / 3 ))
   i_press=$(( ($SLURM_ARRAY_TASK_ID - 1) / 3 % 3 ))
   i_cat=$(( ($SLURM_ARRAY_TASK_ID - 1) % 3 ))

   T=${temperature[$i_temp]}
   P=${pressure[$i_press]}
   CAT=${catalyst[$i_cat]}

   export INPUT="template.in"
   export OUTPUT="results/T${T}_P${P}_${CAT}/"

   # Substitute parameters in the input file
   sed -e "s/TEMPERATURE/$T/g" \
       -e "s/PRESSURE/$P/g" \
       -e "s/CATALYST/$CAT/g" \
       template.in > input_${SLURM_ARRAY_TASK_ID}.in

   job-nanny ./simulate input_${SLURM_ARRAY_TASK_ID}.in

Example 3: Job Array with a Simultaneous Limit
----------------------------------------------

To avoid overloading the system:

.. code-block:: bash
   :caption: limited_array.sh

   #!/bin/bash
   #SBATCH -J limited
   #SBATCH --array=1-1000%50   # Maximum 50 simultaneous jobs
   #SBATCH -n 1
   #SBATCH -t 01:00:00

   # Processing here
   ./my_program input_${SLURM_ARRAY_TASK_ID}.dat

Example 4: Job Array with Dependencies
--------------------------------------

Two-stage processing:

.. code-block:: bash

   # Submit the processing array
   JOBID=$(sbatch --parsable --array=1-100 process_array.sh)

   # Submit the post-processing job (only after all finish)
   sbatch --dependency=afterok:$JOBID post_processing.sh

**post_processing.sh:**

.. code-block:: bash
   :caption: post_processing.sh

   #!/bin/bash
   #SBATCH -J post
   #SBATCH -n 1
   #SBATCH -t 01:00:00

   # Collect all results and generate a summary
   cat results/result_*.dat > all_results.dat
   python generate_statistics.py all_results.dat

Environment Variables
=====================

=========================== ========================= =======
Variable                    Description               Example
--------------------------- ------------------------- -------
``$SLURM_ARRAY_JOB_ID``     Array job ID              123456
``$SLURM_ARRAY_TASK_ID``    Index of the current task 42
``$SLURM_ARRAY_TASK_COUNT`` Total number of tasks     100
``$SLURM_ARRAY_TASK_MIN``   Minimum index             1
``$SLURM_ARRAY_TASK_MAX``   Maximum index             100
=========================== ========================= =======

Monitoring Job Arrays
=====================

View all jobs in the array
--------------------------

.. code-block:: bash

   squeue -u $USER -t PD,R -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

**Example output:**

.. code-block:: text

   JOBID       PARTITION         NAME  USER  ST        TIME  NODES  NODELIST
   2940077_1      medium   submit_job  joao   R  2-06:11:48      1  node032
   2940077_2      medium   submit_job  joao   R  2-04:31:51      1  node050
   2940077_3      medium   submit_job  joao   R  2-03:48:25      1  node040
   2940077_4      medium   submit_job  joao  PD        0:00      1  (Priority)
   2940077_5      medium   submit_job  joao  PD        0:00      1  (Priority)

Cancel a job array
------------------

.. code-block:: bash

   # Cancel a specific task
   scancel 2940077_42

   # Cancel the entire array
   scancel 2940077

   # Cancel all pending tasks in the array
   scancel -t PD 2940077

View details of a task
----------------------

.. code-block:: bash

   scontrol show job 2940077_42

Limitations
===========

- **Maximum jobs per array:** 1000 (current GridUnesp configuration)
- **Maximum simultaneous jobs:** Controlled by ``%N`` in the array
- **Resources per job:** Each array task can have its own requests
- **File names:** Use ``%A`` (array job ID) and ``%a`` (task ID) to avoid conflicts

.. attention::
   Currently, GridUnesp is configured to accept a **maximum of 1000 jobs per array**.


Best Practices
==============

1. **Organize the output files**
   
   .. code-block:: bash

      #SBATCH --output=logs/array_%A_%a.out
      #SBATCH --error=logs/array_%A_%a.err
      
      mkdir -p logs results

2. **Simultaneous job limit**
   
   .. code-block:: bash

      #SBATCH --array=1-1000%50   # Doesn't overload the system

3. **Test with a subset first**
   
   .. code-block:: bash

      #SBATCH --array=1-10   # Test with 10 jobs before 1000

4. **Use meaningful names**
   
   .. code-block:: bash

      #SBATCH -J sweep_temp_pressure

5. **Monitor progress**
   
   .. code-block:: bash

      squeue -u $USER | grep "your_job" | wc -l   # Count active jobs

Comparison: Job Array vs Individual Jobs
========================================

.. list-table:: Comparison: Job Array vs Individual Jobs
   :header-rows: 1
   :widths: 30 35 35

   * - Aspect
     - Job Array
     - Individual Jobs
   * - Submission
     - 1 command
     - N commands
   * - Management
     - Centralized
     - Scattered
   * - Dependencies
     - Easy (afterok:JOBID)
     - Complex
   * - Cancellation
     - 1 command or selective
     - N commands
   * - Job limit
     - 1000 per array
     - 10000 in total
   * - Organization
     - Automatic
     - Manual

.. seealso::

   - :ref:`running_simulations` - Basic submission concepts
   - :ref:`optimizing_performance` - HTC vs HPC strategies
   - :ref:`improving_submission_script` - More script options
   - :ref:`slurm_reference` - Complete SLURM reference