Partitions and Resource Limits

In this section:

GridUnesp organizes computational resources into partitions (queues) to manage job execution efficiently and fairly.

Basic Concepts 

Partition: A set of nodes with similar characteristics and policies
Time limit: The maximum time a job can run
Resource limit: Maximum number of cores, memory, and GPUs per job
Priority policy: Rules that determine execution order

Available Partitions 

GridUnesp Partitions
Partition	Nodes	Maximum Time	Recommended Use
short	All CPU nodes	24 hours	Tests, quick jobs
medium	All CPU nodes	7 days	Medium-length simulations
long	All CPU nodes	30 days	Long simulations
gpu	gpunode001	24 hours	GPU jobs

Note

The short, medium, and long partitions are determined automatically based on the requested time. There is no need to specify them explicitly.

Querying Partitions 

sinfo Command 

# List all partitions
sinfo

# Detailed format
sinfo -o "%20P %5a %10l %6D %6t %N"

Example output:

PARTITION            AVAIL TIMELIMIT  NODES  STATE  NODELIST
short*               up    1-00:00:00 56     mix    node[001-056]
medium               up    7-00:00:00 56     mix    node[001-056]
long                 up    30-00:00:0 56     mix    node[001-056]
gpu                  up    1-00:00:00 1      mix    gpunode001

scontrol Command 

# Details of a specific partition
scontrol show partition short

# List all nodes
scontrol show nodes

Specifying Resources in the Script 

Partition 

# For GPU jobs
#SBATCH --partition=gpu

Note

For the time-based partitions (short/medium/long), no explicit specification is needed. SLURM selects automatically based on the --time value.

Execution Time 

# Format: minutes
#SBATCH -t 30              # 30 minutes (short partition)

# Format: hours:minutes:seconds
#SBATCH -t 12:00:00        # 12 hours (short partition)

# Format: days-hours
#SBATCH -t 2-12:00:00      # 2 days and 12 hours (medium partition)
#SBATCH -t 7-00:00:00      # 7 days (medium partition)
#SBATCH -t 30-00:00:00     # 30 days (long partition)

Important

Time up to 24h → short partition
Time between 24h and 7 days → medium partition
Time between 7 and 30 days → long partition
Time > 30 days → invalid (job will not be accepted)

Nodes and Processes 

# Number of nodes
#SBATCH --nodes=2

# Total processes
#SBATCH --ntasks=52

# Processes per node
#SBATCH --ntasks-per-node=28

# CPUs per process (for threads)
#SBATCH --cpus-per-task=4

Memory 

# Memory per node
#SBATCH --mem=64G

# Memory per CPU
#SBATCH --mem-per-cpu=4G

Warning

Do not use --mem and --mem-per-cpu together. Choose one.

GPUs 

# Number of GPUs
#SBATCH --gres=gpu:1
#SBATCH --gres=gpu:2

# Specific GPU (if more than one type is available)
#SBATCH --gres=gpu:l40s:1

Practical Examples 

Example 1: Serial Job 

serial.sh

#!/bin/bash
#SBATCH -J serial
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=1
#SBATCH --time=04:00:00
#SBATCH --mem=4G

export INPUT="input.dat"
export OUTPUT="output.dat"

job-nanny ./my_program

Example 2: OpenMP Parallel Job 

openmp.sh

#!/bin/bash
#SBATCH -J openmp
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=28
#SBATCH --time=12:00:00
#SBATCH --mem=64G

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
export INPUT="data.dat"
export OUTPUT="results/"

job-nanny ./program_omp

Example 3: MPI Job 

mpi.sh

#!/bin/bash
#SBATCH -J mpi
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=28
#SBATCH --time=48:00:00
#SBATCH --mem-per-cpu=2G

export INPUT="mpi_data/"
export OUTPUT="mpi_results/"

module load openmpi/4.1.5
job-nanny mpirun -np 112 ./program_mpi

Example 4: GPU Job 

gpu.sh

#!/bin/bash
#SBATCH -J gpu
#SBATCH --partition=gpu
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --gres=gpu:2
#SBATCH --cpus-per-task=16
#SBATCH --time=12:00:00
#SBATCH --mem=64G

export INPUT="gpu_data/"
export OUTPUT="gpu_results/"

module load cuda/12.0
job-nanny ./program_cuda

Example 5: Job Array with Concurrency Limit 

array.sh

#!/bin/bash
#SBATCH -J array
#SBATCH --array=1-1000%50
#SBATCH --ntasks=1
#SBATCH --time=01:00:00
#SBATCH --mem=2G

export INPUT="data_${SLURM_ARRAY_TASK_ID}.dat"
export OUTPUT="result_${SLURM_ARRAY_TASK_ID}.dat"

job-nanny ./process

Checking Limits and Usage 

Running Jobs 

# Your jobs
squeue -u $USER

# All jobs
squeue

# Only running jobs
squeue -t RUNNING

# Only pending jobs
squeue -t PENDING

Resources in Use 

# Cluster usage summary
sinfo

# Detailed per-node usage
sinfo -N -o "%16N %8c %8m %8d %6t"

# Available nodes
sinfo -t idle

Why Is My Job Not Starting?

Common reasons for jobs remaining in PENDING state:

Insufficient resources
- All nodes are busy
- Solution: Wait, or request fewer resources
Low priority
- According to the Priority Policy, your Fair Share is low
- Solution: Reduce your usage or wait for the priority reset
Requested resources are infeasible
- Requested more than available (e.g. 100 nodes)
- Solution: Review your resource requests
Invalid requested time
- More than 30 days
- Solution: Reduce to ≤ 30 days

Checking the reason:

squeue -u $USER -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R"

The %R column shows the reason (e.g. “Resources”, “Priority”).

Optimizing Resource Usage 

Request only what you need
- Estimate time and memory accurately
- Use sacct to see actual usage after the job finishes
Smaller jobs have higher priority
- Prefer multiple small jobs over one large one
- Use job arrays for HTC workloads
Avoid submission spikes
- Spread submissions out over time
- Use --array=1-1000%50 to limit concurrent tasks
Monitor your Fair Share
```
sshare -U $USER
```