Job Array

In this section:

Concept
When to Use a Job Array
Basic Syntax
Array Formats
Practical Examples
Environment Variables
Monitoring Job Arrays
Limitations
Best Practices
Comparison: Job Array vs Individual Jobs

A Job Array is a SLURM feature that lets you submit multiple similar jobs with a single script. It is ideal for High Throughput Computing (HTC).

Concept 

A job array is a set of independent jobs, each with its own index. They all share the same submission script but can process different data based on the index.

Analogy: You hire a clothing company (the HPC cluster) and hand over a single pattern (the job script) and a list of parameters (color names or a range of numbers, such as 0 to 10). The company (Slurm) examines the pattern and assigns 8 seamstresses (CPUs) to work (process) simultaneously, each one taking a different parameter (color) from the list using $SLURM_ARRAY_TASK_ID.

When to Use a Job Array 

✅ Ideal for:

Processing many independent files
Parameter sweeps
Monte Carlo simulations
Embarrassingly parallel data analysis
Bootstrapping and cross-validation

❌ Not recommended for:

Jobs with simultaneous dependency on one another
Communication between jobs
Problems that require synchronization

Basic Syntax 

#!/bin/bash
#SBATCH --array=1-100
#SBATCH -n 1
#SBATCH -t 01:00:00

echo "My index is $SLURM_ARRAY_TASK_ID"

Array Formats 

Simple range: --array=1-100 (indices 1 to 100)
Specific list: --array=1,5,10,20,50
With step: --array=1-100:5 (1,6,11,…,96)
With simultaneous limit: --array=1-1000%50 (maximum of 50 jobs running at the same time)

Practical Examples 

Example 1: Processing Multiple Files 

Suppose you have 100 input files named data_1.dat, data_2.dat, …, data_100.dat.

process_array.sh

#!/bin/bash
#SBATCH -J process_data
#SBATCH --array=1-100
#SBATCH -n 1
#SBATCH -t 02:00:00
#SBATCH --mem=4G
#SBATCH --output=logs/array_%A_%a.out
#SBATCH --error=logs/array_%A_%a.err

# Job array variables
# %A = Array Job ID
# %a = Task ID (index)

mkdir -p results/

export INPUT="data_${SLURM_ARRAY_TASK_ID}.dat"
export OUTPUT="results/result_${SLURM_ARRAY_TASK_ID}.dat"

module load python/3.9
job-nanny python process.py $INPUT -o $OUTPUT

Example 2: Parameter Sweep 

To test different parameter combinations:

sweep_array.sh

#!/bin/bash
#SBATCH -J sweep
#SBATCH --array=1-27
#SBATCH -n 1
#SBATCH -t 04:00:00

# Define parameter combinations
temperature=(300 310 320 330 340 350 360 370 380)
pressure=(1.0 1.5 2.0)
catalyst=("Fe" "Ni" "Co")

# Compute indices
i_temp=$(( ($SLURM_ARRAY_TASK_ID - 1) / 3 / 3 ))
i_press=$(( ($SLURM_ARRAY_TASK_ID - 1) / 3 % 3 ))
i_cat=$(( ($SLURM_ARRAY_TASK_ID - 1) % 3 ))

T=${temperature[$i_temp]}
P=${pressure[$i_press]}
CAT=${catalyst[$i_cat]}

export INPUT="template.in"
export OUTPUT="results/T${T}_P${P}_${CAT}/"

# Substitute parameters in the input file
sed -e "s/TEMPERATURE/$T/g" \
    -e "s/PRESSURE/$P/g" \
    -e "s/CATALYST/$CAT/g" \
    template.in > input_${SLURM_ARRAY_TASK_ID}.in

job-nanny ./simulate input_${SLURM_ARRAY_TASK_ID}.in

Example 3: Job Array with a Simultaneous Limit 

To avoid overloading the system:

limited_array.sh

#!/bin/bash
#SBATCH -J limited
#SBATCH --array=1-1000%50   # Maximum 50 simultaneous jobs
#SBATCH -n 1
#SBATCH -t 01:00:00

# Processing here
./my_program input_${SLURM_ARRAY_TASK_ID}.dat

Example 4: Job Array with Dependencies 

Two-stage processing:

# Submit the processing array
JOBID=$(sbatch --parsable --array=1-100 process_array.sh)

# Submit the post-processing job (only after all finish)
sbatch --dependency=afterok:$JOBID post_processing.sh

post_processing.sh:

post_processing.sh

#!/bin/bash
#SBATCH -J post
#SBATCH -n 1
#SBATCH -t 01:00:00

# Collect all results and generate a summary
cat results/result_*.dat > all_results.dat
python generate_statistics.py all_results.dat

Environment Variables 

Variable	Description	Example
`$SLURM_ARRAY_JOB_ID`	Array job ID	123456
`$SLURM_ARRAY_TASK_ID`	Index of the current task	42
`$SLURM_ARRAY_TASK_COUNT`	Total number of tasks	100
`$SLURM_ARRAY_TASK_MIN`	Minimum index	1
`$SLURM_ARRAY_TASK_MAX`	Maximum index	100

Monitoring Job Arrays 

View all jobs in the array 

squeue -u $USER -t PD,R -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

Example output:

JOBID       PARTITION         NAME  USER  ST        TIME  NODES  NODELIST
2940077_1      medium   submit_job  joao   R  2-06:11:48      1  node032
2940077_2      medium   submit_job  joao   R  2-04:31:51      1  node050
2940077_3      medium   submit_job  joao   R  2-03:48:25      1  node040
2940077_4      medium   submit_job  joao  PD        0:00      1  (Priority)
2940077_5      medium   submit_job  joao  PD        0:00      1  (Priority)

Cancel a job array 

# Cancel a specific task
scancel 2940077_42

# Cancel the entire array
scancel 2940077

# Cancel all pending tasks in the array
scancel -t PD 2940077

View details of a task 

scontrol show job 2940077_42

Limitations 

Maximum jobs per array: 1000 (current GridUnesp configuration)
Maximum simultaneous jobs: Controlled by %N in the array
Resources per job: Each array task can have its own requests
File names: Use %A (array job ID) and %a (task ID) to avoid conflicts

Attention

Currently, GridUnesp is configured to accept a maximum of 1000 jobs per array.

Best Practices 

Organize the output files

#SBATCH --output=logs/array_%A_%a.out
#SBATCH --error=logs/array_%A_%a.err

mkdir -p logs results

Simultaneous job limit

#SBATCH --array=1-1000%50   # Doesn't overload the system

Test with a subset first

#SBATCH --array=1-10   # Test with 10 jobs before 1000

Use meaningful names
```
#SBATCH -J sweep_temp_pressure
```

Monitor progress

squeue -u $USER | grep "your_job" | wc -l   # Count active jobs

Comparison: Job Array vs Individual Jobs 

Comparison: Job Array vs Individual Jobs
Aspect	Job Array	Individual Jobs
Submission	1 command	N commands
Management	Centralized	Scattered
Dependencies	Easy (afterok:JOBID)	Complex
Cancellation	1 command or selective	N commands
Job limit	1000 per array	10000 in total
Organization	Automatic	Manual