.. _improving_submission_script:

===============================
Improving the Submission Script
===============================

.. contents:: In this section:
   :local:
   :depth: 2

This section details the construction of robust and efficient submission scripts, incorporating best practices and all the resources available on GridUnesp.

SBATCH Options
==============

SLURM offers dozens of directives to control job execution. The most important ones are below:

.. list-table:: Main SBATCH directives
   :header-rows: 1
   :widths: 25 15 60

   * - Directive
     - Abbr.
     - Description
   * - ``--time``
     - ``-t``
     - Maximum run time (format: ``min``, ``min:sec``, ``hours:min:sec``, ``days-hours``, ``days-hours:min:sec``)
   * - ``--cpus-per-task``
     - ``-c``
     - Number of CPUs per process (for threads/OpenMP)
   * - ``--ntasks``
     - ``-n``
     - Total number of processes (for MPI)
   * - ``--nodes``
     - ``-N``
     - Minimum number of nodes
   * - ``--mem``
     - 
     - Memory per node (e.g., ``16G``, ``32768M``)
   * - ``--mem-per-cpu``
     - 
     - Memory per CPU (e.g., ``4G``)
   * - ``--output``
     - ``-o``
     - File for standard output (stdout)
   * - ``--error``
     - ``-e``
     - File for error output (stderr)
   * - ``--job-name``
     - ``-J``
     - Job name (appears in ``squeue``)
   * - ``--partition``
     - ``-p``
     - Partition (``short``, ``medium``, ``long``, ``gpu``)
   * - ``--gres``
     - 
     - Generic resources, such as GPUs (``--gres=gpu:1``)
   * - ``--mail-type``
     - 
     - When to send email (``BEGIN``, ``END``, ``FAIL``, ``ALL``)
   * - ``--mail-user``
     - 
     - Email address for notifications

**Examples:**

.. code-block:: bash

   #SBATCH -t 24:00:00 -c 4
   #SBATCH --mem=16G
   #SBATCH --output=job_%j.out
   #SBATCH --mail-type=END,FAIL
   #SBATCH --mail-user=joao@unesp.br

.. tip::
   Use ``%j`` in the file name to include the Job ID automatically.

.. _job_nanny_script:

job-nanny Script
================

The **job-nanny** is an essential GridUnesp script that manages data transfer between ``/home/`` and the temporary working areas.

Why use job-nanny?
------------------

1. **Avoids overloading /home/** - Direct access to /home/ during execution degrades performance
2. **Optimizes performance** - Uses local disks (``/tmp/`` or ``/store/``) that are much faster
3. **Ensures persistence** - Copies results back at the end
4. **Automatic checkpoints** - Saves progress periodically

.. warning::
   The ``job-nanny`` script is recognized when the **gridunesp** module is loaded.
   
   When the user logs into the cluster, this module is loaded automatically.
   
   If you have used the ``module purge`` command, you need to run
  
   .. code-block:: bash

      module load gridunesp
      
   so that ``job-nanny`` is available.

Required Variables
------------------

job-nanny **requires** the definition of two variables:

.. code-block:: bash

   export INPUT="file1.dat file2.dat input_directory/"
   export OUTPUT="result.dat output/"

- **INPUT:** Files/directories that will be copied to the temporary working area
- **OUTPUT:** Files/directories that will be copied back to ``/home/`` at the end

Optional Variables
------------------

.. list-table:: job-nanny variables
   :header-rows: 1
   :widths: 25 25 50

   * - Variable
     - Default
     - Description
   * - ``CHECKPOINT``
     - ``$OUTPUT``
     - Location for saving checkpoints
   * - ``WAIT_CHECKPOINT``
     - 10800 (3h)
     - Interval between checkpoints (seconds)
   * - ``VERBOSE``
     - 0
     - Enables verbose mode (1) for debugging
   * - ``SHARED_FS``
     - "false"
     - If "false", uses ``/tmp/``; if "true", forces use of ``/store/``
   * - ``LARGE_FILES``
     - "false"
     - If "true", forces use of ``/store/`` (for large files)

Examples of Using job-nanny
---------------------------

**Example 1: Simple serial job**

.. code-block:: bash
   :caption: simple_job.sh

   #!/bin/bash
   #SBATCH -J test
   #SBATCH -n 1
   #SBATCH -t 01:00:00

   export INPUT="input.dat"
   export OUTPUT="result.dat"

   module load python/3.9
   job-nanny python script.py input.dat > result.dat

**Example 2: Job with multiple files**

.. code-block:: bash
   :caption: multiple_job.sh

   #!/bin/bash
   #SBATCH -J simulation
   #SBATCH -c 4
   #SBATCH -t 24:00:00

   export INPUT="parameters.in data/ library.so"
   export OUTPUT="output.dat logs/"

   module load gcc/10.2.0
   job-nanny ./program -i parameters.in -o output.dat

**Example 3: Job with large files**

.. code-block:: bash
   :caption: large_job.sh

   #!/bin/bash
   #SBATCH -J big_data
   #SBATCH -n 1
   #SBATCH -t 12:00:00

   export INPUT="large_dataset.bin"
   export OUTPUT="results/"
   export LARGE_FILES="true"      # Forces use of /store/

   job-nanny ./processor large_dataset.bin

**Example 4: Job with frequent checkpoints**

.. code-block:: bash
   :caption: long_job.sh

   #!/bin/bash
   #SBATCH -J long_job
   #SBATCH -n 8
   #SBATCH -t 20-00:00:00          # 20 days

   export INPUT="input.dat"
   export OUTPUT="final_result.dat"
   export CHECKPOINT="checkpoints/"
   export WAIT_CHECKPOINT=3600      # Checkpoint every 1 hour

   module load my_software
   job-nanny simulate --restart checkpoints/

.. _job_nanny_parameters:

Choosing Between /tmp/ and /store/
==================================

job-nanny automatically decides where to run:

**Uses /tmp/ (fast, local) when:**

- The job uses **only 1 node** (``-N 1``)
- ``SHARED_FS`` is not set or is ``"false"``
- ``LARGE_FILES`` is not set or is ``"false"``

**Uses /store/ (shared) when:**

- The job uses **multiple nodes** (``-N > 1``)
- ``SHARED_FS="true"``
- ``LARGE_FILES="true"``

.. tip::
   For single-node jobs with very large files (>100 GB), use ``LARGE_FILES="true"`` to force the use of ``/store/``, which has more available space than the ``/tmp/`` of the processing node.

.. _detailed_queue_system:

Queue System (Detailed)
=======================

The specified time automatically determines the partition:

.. list-table:: Time vs partition relationship
   :header-rows: 1
   :widths: 35 20 45

   * - Requested time
     - Partition
     - Maximum processing deadline
   * - Up to 24 hours
     - short
     - 24 hours
   * - Up to 24 hours (specifying ``-p gpu``)
     - gpu
     - 24 hours (run on the GPU server)
   * - Between 24h and 7 days
     - medium
     - 7 days
   * - Between 7 and 30 days
     - long
     - 30 days
   * - Above 30 days
     - (invalid)
     - Job will not be accepted

.. important::
   Always specify the most precise time possible. Jobs with shorter times:
   
   - Have **higher priority** in the queue
   - Can "fit" into windows of idle resources
   - Increase cluster efficiency for everyone

Examples of Time Configuration:

.. code-block:: bash

   # 30 minutes
   #SBATCH -t 30:00

   # 12 hours and 30 minutes
   #SBATCH -t 12:30:00

   # 2 days and 6 hours
   #SBATCH -t 2-06:00:00

   # 30 days (maximum allowed)
   #SBATCH -t 30-00:00:00

.. _detailed_storage_info:

Storage Information (Detailed)
==============================

GridUnesp has three main storage areas, each with specific characteristics:

.. list-table:: Comparison of partitions
   :header-rows: 1
   :widths: 15 20 20 25 20

   * - Partition
     - Capacity
     - Speed
     - Persistence
     - Recommended use
   * - ``/home/``
     - 120 TB
     - Medium
     - Permanent
     - Scripts, code, small data
   * - ``/tmp/``
     - ~180 GB/node
     - Very fast
     - Temporary (only during the job)
     - I/O intensive, temporary files
   * - ``/store/``
     - 7 TB
     - Medium-low
     - Temporary
     - Large files, shared data

.. _tmp_partition_basic:

/tmp/ Partition
---------------

``/tmp/`` is a local directory on each processing node. Its main characteristics:

- **Speed:** Very high (local SSD/HD disk)
- **Space:** Limited (~180 GB per node)
- **Scope:** Visible only on the node where the job is running
- **Persistence:** **TEMPORARY** - files are removed at the end of the job

.. raw:: html

   <video width="640" height="360" controls>
     <source src="../_static/Access_to_tmp.mp4" type="video/mp4">
   </video>


**When to use /tmp/:**

- Jobs that do a lot of disk reading/writing
- Temporary data that can be recreated
- Processing that fits in the available space
- **Single-node** jobs (default)

**Example of explicit use of /tmp/ (it is already the default):**

.. code-block:: bash

   #!/bin/bash
   #SBATCH -J tmp_job
   #SBATCH -N 1
   #SBATCH -n 28
   #SBATCH -t 24:00:00

   export INPUT="input_data/"
   export OUTPUT="results/"

   # job-nanny will use /tmp/ automatically (1 node, no flags)
   job-nanny ./program

.. _store_partition_basic:

/store/ Partition
-----------------

``/store/`` is a file system shared among all nodes:

- **Speed:** Medium
- **Space:** Large (7 TB)
- **Scope:** Visible on all nodes
- **Persistence:** **TEMPORARY** (data deleted after the job ends)

.. raw:: html

   <video width="640" height="360" controls>
     <source src="../_static/Access_to_store.mp4" type="video/mp4">
   </video>

**When to use /store/:**

- **Multi-node** jobs (MPI)
- Very large input/output files (>100 GB)
- Data that needs to be accessed by several nodes simultaneously

**Example of a multi-node job (uses /store/ automatically):**

.. code-block:: bash

   #!/bin/bash
   #SBATCH -J mpi_job
   #SBATCH -N 4
   #SBATCH -n 112
   #SBATCH -t 72:00:00

   export INPUT="large_data/"
   export OUTPUT="mpi_results/"

   module load openmpi/4.1.5
   job-nanny mpirun -n 112 ./mpi_program

**Example forcing the use of /store/ in a single-node job:**

.. code-block:: bash

   #!/bin/bash
   #SBATCH -J big_job
   #SBATCH -N 1
   #SBATCH -n 28
   #SBATCH -t 48:00:00

   export INPUT="dataset_500GB/"
   export OUTPUT="results/"
   export LARGE_FILES="true"      # Forces use of /store/

   job-nanny ./processor

Important Considerations
========================

1. **Never** run jobs directly on the access server
2. **Always** define INPUT and OUTPUT when using job-nanny
3. **Estimate** the required time correctly (shorter jobs have priority)
4. **Monitor** your jobs with ``squeue`` and ``scontrol``
5. **Clean up** temporary files after completion

.. seealso::

   - :ref:`running_simulations` - Basic concepts
   - :ref:`monitoring_jobs` - Advanced monitoring
   - :ref:`storage_guide` - Complete storage guide
   - :ref:`best_practices` - General recommendations