Using job-nanny

job-nanny is an essential GridUnesp script that manages the efficient transfer of data between the /home/ filesystem and the working areas of the compute nodes (/tmp/ or /store/).

Important

The use of job-nanny is mandatory for all jobs that perform read/write operations on disk.

Why Use job-nanny?

  1. Protects /home/ - Intensive access to /home/ during execution degrades performance for everyone

  2. Optimizes performance - Uses much faster local disks (/tmp/)

  3. Ensures persistence - Copies results back at the end

  4. Automatic checkpoints - Saves progress periodically

  5. Automatic cleanup - Removes temporary files after the job

How It Works

Workflow

1. Job submitted with sbatch
2. job-nanny copies INPUT from /home/ to the work area (/tmp/ or /store/)
3. Program runs in the work area
4. Periodic checkpoints (optional)
5. At the end, OUTPUT is copied back to /home/
6. Work area is cleaned up

Required Variables

job-nanny requires two environment variables to be defined:

INPUT

List of files and directories that will be copied from /home/ to the work area.

export INPUT="file1.dat file2.dat input_directory/"

Examples:

# A single file
export INPUT="input.dat"

# Multiple files
export INPUT="input.dat parameters.txt"

# Files and directories
export INPUT="input.dat data_folder/ library.so"

# Using wildcards (use with care!)
export INPUT="*.dat data/"

Tip

If there is more than one input file/directory name, the names must be separated by a space.

OUTPUT

List of files and directories that will be copied from the work area back to /home/.

export OUTPUT="result.dat output/"

Examples:

# A specific output file
export OUTPUT="result.out"

# Multiple files and directories
export OUTPUT="result.out logs/ plots/"

# All files (be careful about size!)
export OUTPUT="*"

# Files matching a specific pattern
export OUTPUT="*.log *.dat results/"

Warning

Caution when using wildcards (*)

Using * in INPUT/OUTPUT may copy many unnecessary files and take a long time. Be as specific as possible.

Optional Variables

Optional job-nanny variables

Variable

Default

Description

CHECKPOINT

$OUTPUT

Location to save checkpoints (can differ from OUTPUT)

WAIT_CHECKPOINT

10800 (3h)

Interval between checkpoints in seconds

SHARED_FS

“false”

Forces use of /store/ (for large files or multi-node jobs)

LARGE_FILES

“false”

Indicates large files (>100 GB), forces use of /store/

VERBOSE

0

Enables verbose mode (1) for debugging

CHECKPOINT_FUNC

Custom checkpoint function (advanced use)

Choosing Between /tmp/ and /store/

job-nanny automatically decides where to run:

Uses /tmp/ (fast, local) when:

  • Job uses only 1 node (-N 1)

  • SHARED_FS is not set or is "false"

  • LARGE_FILES is not set or is "false"

Uses /store/ (shared) when:

  • Job uses multiple nodes (-N > 1)

  • SHARED_FS="true"

  • LARGE_FILES="true"

Tip

For single-node jobs with very large files (>100 GB), use LARGE_FILES="true" to force the use of /store/, which has more available space.

Usage Examples

Example 1: Simple Serial Job

job_simple.sh
#!/bin/bash
#SBATCH -J simple
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -t 01:00:00
#SBATCH --mem=4G

export INPUT="input.dat"
export OUTPUT="result.dat"

module load python/3.9
job-nanny python process.py input.dat > result.dat

Example 2: Job with Multiple Files

job_multiple.sh
#!/bin/bash
#SBATCH -J multiple
#SBATCH -N 1
#SBATCH -c 4
#SBATCH -t 06:00:00

export INPUT="parameters.in data/ libraries/"
export OUTPUT="output.dat logs/"

module load my_app
job-nanny ./simulate -i parameters.in -o output.dat

Example 3: Job with Large Files

job_large.sh
#!/bin/bash
#SBATCH -J large
#SBATCH -N 1
#SBATCH -n 28
#SBATCH -t 48:00:00

export INPUT="dataset_500GB.bin"
export OUTPUT="results/"
export LARGE_FILES="true"      # Forces use of /store/

job-nanny srun ./processor dataset_500GB.bin

Example 4: MPI Job (Multiple Nodes)

job_mpi.sh
#!/bin/bash
#SBATCH -J mpi
#SBATCH -N 4
#SBATCH --ntasks-per-node=28
#SBATCH -t 24:00:00

export INPUT="mpi_data/"
export OUTPUT="mpi_results/"

module load openmpi/4.1.5
job-nanny mpirun -np $SLURM_NTASKS ./program_mpi

Note

Jobs with -N > 1 use /store/ automatically, even without SHARED_FS="true" or LARGE_FILES="true".

Example 5: Job with Frequent Checkpoints

job_checkpoint.sh
#!/bin/bash
#SBATCH -J long
#SBATCH -N 1
#SBATCH -n 28
#SBATCH -t 20-00:00:00          # 20 days

export INPUT="input.dat"
export OUTPUT="final_result.dat"
export CHECKPOINT="checkpoints/"
export WAIT_CHECKPOINT=3600      # Checkpoint every 1 hour

module load my_software
job-nanny simulate --restart checkpoints/

Example 6: Debug Job

job_debug.sh
#!/bin/bash
#SBATCH -J debug
#SBATCH -N 1
#SBATCH -n 4
#SBATCH -t 01:00:00

export INPUT="test_input.dat"
export OUTPUT="test_output.dat"
export VERBOSE=1                  # Verbose mode

module load my_app
job-nanny ./program test_input.dat

job-nanny Log File

job-nanny generates detailed output in the SLURM log file (slurm-JOBID.out):

Hostname node045
Basedir /tmp
Copying /home/john/input.dat -> /tmp/john/12345/
Executing ./program input.dat
Return Code 0
Copying /tmp/john/12345/result.dat -> /home/john/
Removing files from /tmp/john/12345/

With VERBOSE=1, more details are shown:

Hostname node045
Basedir /tmp
Creating directory /tmp/john/12345/
Copying file1.dat
Copying file2.dat
Copying directory/
Executing ./program
Checkpoint thread started (interval: 10800s)
[Checkpoint] Saving at 18:30:00
[Checkpoint] Completed
Return Code 0
Stopping checkpoint thread
Copying result.dat
Copying logs/
Removing work directory

Common Errors and Solutions

Error: “INPUT is missing”

Cause: The INPUT variable was not defined.

Solution:

export INPUT="your_files.dat"

Error: “OUTPUT is missing”

Cause: The OUTPUT variable was not defined.

Solution:

export OUTPUT="your_results.dat"

Error: “No space left on device”

Cause: Full disk (usually /tmp/).

Solutions:

  1. Reduce the size of INPUT files

  2. Use LARGE_FILES="true" to switch to /store/

  3. Clean up temporary files during execution

OUTPUT Files Do Not Appear in /home/

Possible causes:

  1. Wrong names in OUTPUT

    # If your program generates "output.dat"
export OUTPUT="output.dat"   # Correct
export OUTPUT="result.dat"   # Wrong!

  2. Job cancelled before copying

    • Use more frequent checkpoints (smaller WAIT_CHECKPOINT)

  3. File permissions

    • Check whether the program has permission to create the files

Error: “Permission denied” on /store/

Cause: Incorrect directory permissions.

Solution:

# Check permissions
ls -la /store/$USER

# Fix if necessary
chmod 755 /store/$USER

Frequently Asked Questions

Q: Is it mandatory to use job-nanny?

A: Yes, for any job that does significant I/O. Jobs that access /home/ directly during execution may be cancelled by the team.

Q: Can I run without job-nanny?

A: Technically yes, but it is not recommended. Performance will be worse and you may overload the system.

Q: Does job-nanny work with MPI?

A: Yes! Use job-nanny mpirun -np $SLURM_NTASKS ./program_mpi.

Q: How do I use libraries installed in /home/?

A: Include them in INPUT and configure LD_LIBRARY_PATH:

export INPUT="program/libs/library.lib"
export LD_LIBRARY_PATH=/path/to/program/libs:$LD_LIBRARY_PATH
job-nanny ./program

Q: Can I use job-nanny with containers?

A: Yes:

export INPUT="container.sif script.py"
export OUTPUT="results/"
job-nanny apptainer exec container.sif python script.py

See also