Optimizing Performance

In this section:

Why Optimize?
Factors to Consider
Practical Optimization Example
Optimization Strategies
HTC: “Breaking Up” Large Inputs (HTC)
Trade-offs Between /tmp/ and /store/
Optimization Checklist

Before submitting jobs for definitive processing, it is essential to carry out optimization studies. The goal is to find the ideal balance between requested resources, queue wait time, and run time.

Why Optimize?

Verify that the software installation was successful
Discover the minimum resources required for good performance
Increase priority by requesting only what is needed
Reduce wait time in the queue
Get results faster with efficient use of resources

Factors to Consider 

Optimization variables
Factor	Impact
Number of CPUs	More CPUs can speed things up, but not always linearly
Available memory	Affects processing capacity and avoids swapping
Number of nodes	Distributing across nodes can introduce communication latency
Requested time	Shorter jobs have higher priority
Data I/O (Read/Write) location	`/tmp/` is much faster than `/store/`

Practical Optimization Example 

Let’s follow the user Spock optimizing his program “warp_4.py”.

Scenario: The program processes input files and produces results. Spock wants to find the ideal configuration.

Test 1: 24 processes on 12 nodes (1 CPU per process)

test1.sh

#!/bin/bash
#SBATCH -t 24:00:00
#SBATCH -n 24
#SBATCH -N 12
#SBATCH -c 1

export INPUT="energy.in fuel.in warp_4.py"
export OUTPUT="light-speed.data"

module load intel/compilers
module load anaconda3
source activate startrek

job-nanny srun python warp_4.py --input=energy.in,fuel.in

Results:

Run time: 13h55min
Queue wait: 10h
Total time to result: 23h55min

Important

The srun command is a tool used in the Slurm Workload Manager (a task scheduler commonly used on high-performance computers and supercomputers). It is used to request computational resources (such as CPUs, GPUs, and memory) and run parallel tasks on those resources.

Test 2: 24 processes on 8 nodes (1 CPU per process)

test2.sh

#!/bin/bash
#SBATCH -t 24:00:00
#SBATCH -n 24
#SBATCH -N 8
#SBATCH -c 1

Results:

Run time: 11h15min
Queue wait: 6h
Total time to result: 17h15min ✓ Best so far

Test 3: 24 processes on 12 nodes (2 CPUs per process)

test3.sh

#!/bin/bash
#SBATCH -t 24:00:00
#SBATCH -n 24
#SBATCH -N 12
#SBATCH -c 2      # 2 CPUs per process = 48 CPUs total

Results:

Run time: 6h30min (faster!)
Queue wait: 25h (much longer wait)
Total time to result: 31h30min

Test 4: 24 processes on 1 node (2 CPUs per process)

test4.sh

#!/bin/bash
#SBATCH -t 24:00:00
#SBATCH -n 24
#SBATCH -N 1
#SBATCH -c 2      # 2 CPUs per process = 48 CPUs total

Results:

Run time: 3h (excellent!)
Queue wait: 72h (very long)
Total time to result: 75h

Comparative Table 

Test results
Test	Processes	Nodes	CPUs/Proc	Total CPUs	Wait	Run	Total time
1	24	12	1	24	10:00	13:55	23:55
2	24	8	1	24	06:00	11:15	17:15
3	24	12	2	48	25:00	06:30	31:30
4	24	1	2	48	72:00	03:00	75:00

Spock’s conclusion: Test 2 offers the best balance between wait and run time (17h15min total).

Optimization Strategies 

Start with minimum resources

# Initial test with few resources
#SBATCH -n 1
#SBATCH -t 01:00:00
#SBATCH --mem=2G

Increase gradually

# Next test
#SBATCH -n 4
#SBATCH -t 02:00:00
#SBATCH --mem=8G

Monitor actual usage

# After the job finishes
sacct -j JOBID --format=JobID,MaxRSS,Elapsed,CPUTime

Adjust based on the data
- If MaxRSS is much smaller than the requested memory, reduce the memory
- If Elapsed is much shorter than the requested time, reduce the time
- If CPU time is much greater than Elapsed, there is inefficient parallelization

HTC: “Breaking Up” Large Inputs (HTC)

GridUnesp can be used in two ways:

HPC (High Performance Computing): Many threads for a single job
HTC (High Throughput Computing): Many small jobs processing parts of a large problem

HTC example:

Instead of 1 job processing a 10 GB file:

# Single large job
./process complete_data_10GB.dat

you can split it into 100 jobs each processing 100 MB files:

job_array_htc.sh

#!/bin/bash
#SBATCH --array=1-100
#SBATCH -n 1
#SBATCH -t 01:00:00
#SBATCH --mem=2G

export INPUT="data_part_${SLURM_ARRAY_TASK_ID}.dat"
export OUTPUT="result_part_${SLURM_ARRAY_TASK_ID}.dat"

job-nanny ./process $INPUT > $OUTPUT

Advantages of HTC:

Each small job starts faster
Better use of idle resources
Lower impact from failures (only a few sub-jobs are lost)
Nearly linear scalability

Trade-offs Between /tmp/ and /store/

Performance comparison
Configuration	I/O location	Speed	When to use
1 node, no flags	`/tmp/`	⚡⚡⚡ Very fast	I/O intensive, data <100GB
1 node, `LARGE_FILES="true"`	`/store/`	⚡ Medium	Data >100GB, 1 node
Multiple nodes	`/store/`	⚡ Medium	MPI jobs, shared data

Recommendation:

Whenever possible, use 1 node and /tmp/ for I/O-intensive work
For very large data (>100GB), use export LARGE_FILES="true" in the submission script
For MPI jobs, accept the latency of /store/ in exchange for scalability

Optimization Checklist 

Before submitting your definitive job:

[__] I tested with a small sample of the data
[__] I monitored actual usage with sacct
[__] I adjusted memory to the real value (with a safety margin)
[__] I adjusted time to the real value (with a safety margin)
[__] I chose the ideal number of CPUs/processes
[__] I decided between 1 node (/tmp/) or multiple nodes (/store/)
[__] I considered splitting into a Job Array (HTC)
[__] I checked my group’s Fair Share (Priority Policy)