Improving the Submission Script

In this section:

SBATCH Options
job-nanny Script
Choosing Between /tmp/ and /store/
Queue System (Detailed)
Storage Information (Detailed)
- /tmp/ Partition
- /store/ Partition
Important Considerations

This section details the construction of robust and efficient submission scripts, incorporating best practices and all the resources available on GridUnesp.

SBATCH Options 

SLURM offers dozens of directives to control job execution. The most important ones are below:

Main SBATCH directives
Directive	Abbr.	Description
`--time`	`-t`	Maximum run time (format: `min`, `min:sec`, `hours:min:sec`, `days-hours`, `days-hours:min:sec`)
`--cpus-per-task`	`-c`	Number of CPUs per process (for threads/OpenMP)
`--ntasks`	`-n`	Total number of processes (for MPI)
`--nodes`	`-N`	Minimum number of nodes
`--mem`		Memory per node (e.g., `16G`, `32768M`)
`--mem-per-cpu`		Memory per CPU (e.g., `4G`)
`--output`	`-o`	File for standard output (stdout)
`--error`	`-e`	File for error output (stderr)
`--job-name`	`-J`	Job name (appears in `squeue`)
`--partition`	`-p`	Partition (`short`, `medium`, `long`, `gpu`)
`--gres`		Generic resources, such as GPUs (`--gres=gpu:1`)
`--mail-type`		When to send email (`BEGIN`, `END`, `FAIL`, `ALL`)
`--mail-user`		Email address for notifications

Examples:

#SBATCH -t 24:00:00 -c 4
#SBATCH --mem=16G
#SBATCH --output=job_%j.out
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=joao@unesp.br

Tip

Use %j in the file name to include the Job ID automatically.

job-nanny Script 

The job-nanny is an essential GridUnesp script that manages data transfer between /home/ and the temporary working areas.

Why use job-nanny?

Avoids overloading /home/ - Direct access to /home/ during execution degrades performance
Optimizes performance - Uses local disks (/tmp/ or /store/) that are much faster
Ensures persistence - Copies results back at the end
Automatic checkpoints - Saves progress periodically

Warning

The job-nanny script is recognized when the gridunesp module is loaded.

When the user logs into the cluster, this module is loaded automatically.

If you have used the module purge command, you need to run

module load gridunesp

so that job-nanny is available.

Required Variables 

job-nanny requires the definition of two variables:

export INPUT="file1.dat file2.dat input_directory/"
export OUTPUT="result.dat output/"

INPUT: Files/directories that will be copied to the temporary working area
OUTPUT: Files/directories that will be copied back to /home/ at the end

Optional Variables 

job-nanny variables
Variable	Default	Description
`CHECKPOINT`	`$OUTPUT`	Location for saving checkpoints
`WAIT_CHECKPOINT`	10800 (3h)	Interval between checkpoints (seconds)
`VERBOSE`	0	Enables verbose mode (1) for debugging
`SHARED_FS`	“false”	If “false”, uses `/tmp/`; if “true”, forces use of `/store/`
`LARGE_FILES`	“false”	If “true”, forces use of `/store/` (for large files)

Examples of Using job-nanny 

Example 1: Simple serial job

simple_job.sh

#!/bin/bash
#SBATCH -J test
#SBATCH -n 1
#SBATCH -t 01:00:00

export INPUT="input.dat"
export OUTPUT="result.dat"

module load python/3.9
job-nanny python script.py input.dat > result.dat

Example 2: Job with multiple files

multiple_job.sh

#!/bin/bash
#SBATCH -J simulation
#SBATCH -c 4
#SBATCH -t 24:00:00

export INPUT="parameters.in data/ library.so"
export OUTPUT="output.dat logs/"

module load gcc/10.2.0
job-nanny ./program -i parameters.in -o output.dat

Example 3: Job with large files

large_job.sh

#!/bin/bash
#SBATCH -J big_data
#SBATCH -n 1
#SBATCH -t 12:00:00

export INPUT="large_dataset.bin"
export OUTPUT="results/"
export LARGE_FILES="true"      # Forces use of /store/

job-nanny ./processor large_dataset.bin

Example 4: Job with frequent checkpoints

long_job.sh

#!/bin/bash
#SBATCH -J long_job
#SBATCH -n 8
#SBATCH -t 20-00:00:00          # 20 days

export INPUT="input.dat"
export OUTPUT="final_result.dat"
export CHECKPOINT="checkpoints/"
export WAIT_CHECKPOINT=3600      # Checkpoint every 1 hour

module load my_software
job-nanny simulate --restart checkpoints/

Choosing Between /tmp/ and /store/

job-nanny automatically decides where to run:

Uses /tmp/ (fast, local) when:

The job uses only 1 node (-N 1)
SHARED_FS is not set or is "false"
LARGE_FILES is not set or is "false"

Uses /store/ (shared) when:

The job uses multiple nodes (-N > 1)
SHARED_FS="true"
LARGE_FILES="true"

Tip

For single-node jobs with very large files (>100 GB), use LARGE_FILES="true" to force the use of /store/, which has more available space than the /tmp/ of the processing node.

Queue System (Detailed)

The specified time automatically determines the partition:

Time vs partition relationship
Requested time	Partition	Maximum processing deadline
Up to 24 hours	short	24 hours
Up to 24 hours (specifying `-p gpu`)	gpu	24 hours (run on the GPU server)
Between 24h and 7 days	medium	7 days
Between 7 and 30 days	long	30 days
Above 30 days	(invalid)	Job will not be accepted

Important

Always specify the most precise time possible. Jobs with shorter times:

Have higher priority in the queue
Can “fit” into windows of idle resources
Increase cluster efficiency for everyone

Examples of Time Configuration:

# 30 minutes
#SBATCH -t 30:00

# 12 hours and 30 minutes
#SBATCH -t 12:30:00

# 2 days and 6 hours
#SBATCH -t 2-06:00:00

# 30 days (maximum allowed)
#SBATCH -t 30-00:00:00

Storage Information (Detailed)

GridUnesp has three main storage areas, each with specific characteristics:

Comparison of partitions
Partition	Capacity	Speed	Persistence	Recommended use
`/home/`	120 TB	Medium	Permanent	Scripts, code, small data
`/tmp/`	~180 GB/node	Very fast	Temporary (only during the job)	I/O intensive, temporary files
`/store/`	7 TB	Medium-low	Temporary	Large files, shared data

/tmp/ Partition 

/tmp/ is a local directory on each processing node. Its main characteristics:

Speed: Very high (local SSD/HD disk)
Space: Limited (~180 GB per node)
Scope: Visible only on the node where the job is running
Persistence: TEMPORARY - files are removed at the end of the job

When to use /tmp/:

Jobs that do a lot of disk reading/writing
Temporary data that can be recreated
Processing that fits in the available space
Single-node jobs (default)

Example of explicit use of /tmp/ (it is already the default):

#!/bin/bash
#SBATCH -J tmp_job
#SBATCH -N 1
#SBATCH -n 28
#SBATCH -t 24:00:00

export INPUT="input_data/"
export OUTPUT="results/"

# job-nanny will use /tmp/ automatically (1 node, no flags)
job-nanny ./program

/store/ Partition 

/store/ is a file system shared among all nodes:

Speed: Medium
Space: Large (7 TB)
Scope: Visible on all nodes
Persistence: TEMPORARY (data deleted after the job ends)

When to use /store/:

Multi-node jobs (MPI)
Very large input/output files (>100 GB)
Data that needs to be accessed by several nodes simultaneously

Example of a multi-node job (uses /store/ automatically):

#!/bin/bash
#SBATCH -J mpi_job
#SBATCH -N 4
#SBATCH -n 112
#SBATCH -t 72:00:00

export INPUT="large_data/"
export OUTPUT="mpi_results/"

module load openmpi/4.1.5
job-nanny mpirun -n 112 ./mpi_program

Example forcing the use of /store/ in a single-node job:

#!/bin/bash
#SBATCH -J big_job
#SBATCH -N 1
#SBATCH -n 28
#SBATCH -t 48:00:00

export INPUT="dataset_500GB/"
export OUTPUT="results/"
export LARGE_FILES="true"      # Forces use of /store/

job-nanny ./processor

Important Considerations 

Never run jobs directly on the access server
Always define INPUT and OUTPUT when using job-nanny
Estimate the required time correctly (shorter jobs have priority)
Monitor your jobs with squeue and scontrol
Clean up temporary files after completion