.. _optimizing_performance:

======================
Optimizing Performance
======================

.. contents:: In this section:
   :local:
   :depth: 2

Before submitting jobs for definitive processing, it is essential to carry out optimization studies. The goal is to find the ideal balance between requested resources, queue wait time, and run time.

Why Optimize?
=============

1. **Verify that the software installation was successful**
2. **Discover the minimum resources required** for good performance
3. **Increase priority** by requesting only what is needed
4. **Reduce wait time** in the queue
5. **Get results faster** with efficient use of resources

Factors to Consider
===================

.. list-table:: Optimization variables
   :header-rows: 1
   :widths: 35 65

   * - Factor
     - Impact
   * - Number of CPUs
     - More CPUs can speed things up, but not always linearly
   * - Available memory
     - Affects processing capacity and avoids swapping
   * - Number of nodes
     - Distributing across nodes can introduce communication latency
   * - Requested time
     - Shorter jobs have higher priority
   * - Data I/O (Read/Write) location
     - ``/tmp/`` is much faster than ``/store/``

Practical Optimization Example
==============================

Let's follow the user Spock optimizing his program "warp_4.py".

**Scenario:** The program processes input files and produces results. Spock wants to find the ideal configuration.

Test 1: 24 processes on 12 nodes (1 CPU per process)
----------------------------------------------------

.. code-block:: bash
   :caption: test1.sh

   #!/bin/bash
   #SBATCH -t 24:00:00
   #SBATCH -n 24
   #SBATCH -N 12
   #SBATCH -c 1

   export INPUT="energy.in fuel.in warp_4.py"
   export OUTPUT="light-speed.data"

   module load intel/compilers
   module load anaconda3
   source activate startrek

   job-nanny srun python warp_4.py --input=energy.in,fuel.in

**Results:**

- Run time: 13h55min
- Queue wait: 10h
- **Total time to result: 23h55min**

.. important::
   The ``srun`` command is a tool used in the `Slurm Workload Manager <https://slurm.schedmd.com/srun.html>`_ (a task scheduler commonly used on high-performance computers and supercomputers). It is used to request computational resources (such as CPUs, GPUs, and memory) and run parallel tasks on those resources.

Test 2: 24 processes on 8 nodes (1 CPU per process)
---------------------------------------------------

.. code-block:: bash
   :caption: test2.sh

   #!/bin/bash
   #SBATCH -t 24:00:00
   #SBATCH -n 24
   #SBATCH -N 8
   #SBATCH -c 1

**Results:**

- Run time: 11h15min
- Queue wait: 6h
- **Total time to result: 17h15min** ✓ Best so far

Test 3: 24 processes on 12 nodes (2 CPUs per process)
-----------------------------------------------------

.. code-block:: bash
   :caption: test3.sh

   #!/bin/bash
   #SBATCH -t 24:00:00
   #SBATCH -n 24
   #SBATCH -N 12
   #SBATCH -c 2      # 2 CPUs per process = 48 CPUs total

**Results:**

- Run time: 6h30min (faster!)
- Queue wait: 25h (much longer wait)
- **Total time to result: 31h30min**

Test 4: 24 processes on 1 node (2 CPUs per process)
---------------------------------------------------

.. code-block:: bash
   :caption: test4.sh

   #!/bin/bash
   #SBATCH -t 24:00:00
   #SBATCH -n 24
   #SBATCH -N 1
   #SBATCH -c 2      # 2 CPUs per process = 48 CPUs total

**Results:**

- Run time: 3h (excellent!)
- Queue wait: 72h (very long)
- **Total time to result: 75h**

Comparative Table
-----------------

.. list-table:: Test results
   :header-rows: 1
   :widths: 10 15 10 15 15 10 10 15

   * - Test
     - Processes
     - Nodes
     - CPUs/Proc
     - Total CPUs
     - Wait
     - Run
     - Total time
   * - 1
     - 24
     - 12
     - 1
     - 24
     - 10:00
     - 13:55
     - 23:55
   * - 2
     - 24
     - 8
     - 1
     - 24
     - 06:00
     - 11:15
     - 17:15
   * - 3
     - 24
     - 12
     - 2
     - 48
     - 25:00
     - 06:30
     - 31:30
   * - 4
     - 24
     - 1
     - 2
     - 48
     - 72:00
     - 03:00
     - 75:00

**Spock's conclusion:** Test 2 offers the best balance between wait and run time (17h15min total).

Optimization Strategies
=======================

1. **Start with minimum resources**
   
   .. code-block:: bash

      # Initial test with few resources
      #SBATCH -n 1
      #SBATCH -t 01:00:00
      #SBATCH --mem=2G

2. **Increase gradually**
   
   .. code-block:: bash

      # Next test
      #SBATCH -n 4
      #SBATCH -t 02:00:00
      #SBATCH --mem=8G

3. **Monitor actual usage**
   
   .. code-block:: bash

      # After the job finishes
      sacct -j JOBID --format=JobID,MaxRSS,Elapsed,CPUTime

4. **Adjust based on the data**
   
   - If MaxRSS is much smaller than the requested memory, reduce the memory
   - If Elapsed is much shorter than the requested time, reduce the time
   - If CPU time is much greater than Elapsed, there is inefficient parallelization

.. _breaking_large_inputs:

HTC: "Breaking Up" Large Inputs (HTC)
=====================================

GridUnesp can be used in two ways:

- **HPC (High Performance Computing):** Many threads for a single job
- **HTC (High Throughput Computing):** Many small jobs processing parts of a large problem

**HTC example:**

Instead of 1 job processing a 10 GB file:

.. code-block:: bash

   # Single large job
   ./process complete_data_10GB.dat

you can split it into 100 jobs each processing 100 MB files:

.. code-block:: bash
   :caption: job_array_htc.sh

   #!/bin/bash
   #SBATCH --array=1-100
   #SBATCH -n 1
   #SBATCH -t 01:00:00
   #SBATCH --mem=2G

   export INPUT="data_part_${SLURM_ARRAY_TASK_ID}.dat"
   export OUTPUT="result_part_${SLURM_ARRAY_TASK_ID}.dat"

   job-nanny ./process $INPUT > $OUTPUT

**Advantages of HTC:**

- Each small job starts faster
- Better use of idle resources
- Lower impact from failures (only a few sub-jobs are lost)
- Nearly linear scalability

Trade-offs Between /tmp/ and /store/
====================================

.. list-table:: Performance comparison
   :header-rows: 1
   :widths: 25 25 25 25

   * - Configuration
     - I/O location
     - Speed
     - When to use
   * - 1 node, no flags
     - ``/tmp/``
     - ⚡⚡⚡ Very fast
     - I/O intensive, data <100GB
   * - 1 node, ``LARGE_FILES="true"``
     - ``/store/``
     - ⚡ Medium
     - Data >100GB, 1 node
   * - Multiple nodes
     - ``/store/``
     - ⚡ Medium
     - MPI jobs, shared data

**Recommendation:**

1. **Whenever possible, use 1 node and /tmp/** for I/O-intensive work
2. **For very large data (>100GB)**, use ``export LARGE_FILES="true"`` in the submission script
3. **For MPI jobs**, accept the latency of /store/ in exchange for scalability

Optimization Checklist
======================

Before submitting your definitive job:

- [__] I tested with a small sample of the data
- [__] I monitored actual usage with ``sacct``
- [__] I adjusted memory to the real value (with a safety margin)
- [__] I adjusted time to the real value (with a safety margin)
- [__] I chose the ideal number of CPUs/processes
- [__] I decided between 1 node (/tmp/) or multiple nodes (/store/)
- [__] I considered splitting into a :ref:`job_array` (HTC)
- [__] I checked my group's Fair Share (:ref:`priority_policy`)

.. seealso::

   - :ref:`priority_policy` - How priority affects your jobs
   - :ref:`shared_memory` - OpenMP and threads
   - :ref:`distributed_memory` - MPI and distributed processes
   - :ref:`job_array` - Multiple jobs for HTC
   - :ref:`storage_guide` - Details on /tmp/ and /store/