.. _partitions_limits: ============================== Partitions and Resource Limits ============================== .. contents:: In this section: :local: :depth: 2 GridUnesp organizes computational resources into partitions (queues) to manage job execution efficiently and fairly. Basic Concepts ============== - **Partition:** A set of nodes with similar characteristics and policies - **Time limit:** The maximum time a job can run - **Resource limit:** Maximum number of cores, memory, and GPUs per job - **Priority policy:** Rules that determine execution order Available Partitions ==================== .. list-table:: GridUnesp Partitions :header-rows: 1 :widths: 15 30 20 35 * - Partition - Nodes - Maximum Time - Recommended Use * - **short** - All CPU nodes - 24 hours - Tests, quick jobs * - **medium** - All CPU nodes - 7 days - Medium-length simulations * - **long** - All CPU nodes - 30 days - Long simulations * - **gpu** - gpunode001 - 24 hours - GPU jobs .. note:: The **short**, **medium**, and **long** partitions are determined automatically based on the requested time. There is no need to specify them explicitly. Querying Partitions =================== sinfo Command ------------- .. code-block:: bash # List all partitions sinfo # Detailed format sinfo -o "%20P %5a %10l %6D %6t %N" **Example output:** .. code-block:: text PARTITION AVAIL TIMELIMIT NODES STATE NODELIST short* up 1-00:00:00 56 mix node[001-056] medium up 7-00:00:00 56 mix node[001-056] long up 30-00:00:0 56 mix node[001-056] gpu up 1-00:00:00 1 mix gpunode001 scontrol Command ---------------- .. code-block:: bash # Details of a specific partition scontrol show partition short # List all nodes scontrol show nodes Specifying Resources in the Script ================================== Partition --------- .. code-block:: bash # For GPU jobs #SBATCH --partition=gpu .. note:: For the time-based partitions (short/medium/long), no explicit specification is needed. SLURM selects automatically based on the ``--time`` value. Execution Time -------------- .. code-block:: bash # Format: minutes #SBATCH -t 30 # 30 minutes (short partition) # Format: hours:minutes:seconds #SBATCH -t 12:00:00 # 12 hours (short partition) # Format: days-hours #SBATCH -t 2-12:00:00 # 2 days and 12 hours (medium partition) #SBATCH -t 7-00:00:00 # 7 days (medium partition) #SBATCH -t 30-00:00:00 # 30 days (long partition) .. important:: - Time up to 24h → **short** partition - Time between 24h and 7 days → **medium** partition - Time between 7 and 30 days → **long** partition - Time > 30 days → **invalid** (job will not be accepted) Nodes and Processes ------------------- .. code-block:: bash # Number of nodes #SBATCH --nodes=2 # Total processes #SBATCH --ntasks=52 # Processes per node #SBATCH --ntasks-per-node=28 # CPUs per process (for threads) #SBATCH --cpus-per-task=4 Memory ------ .. code-block:: bash # Memory per node #SBATCH --mem=64G # Memory per CPU #SBATCH --mem-per-cpu=4G .. warning:: Do not use ``--mem`` and ``--mem-per-cpu`` together. Choose one. GPUs ---- .. code-block:: bash # Number of GPUs #SBATCH --gres=gpu:1 #SBATCH --gres=gpu:2 # Specific GPU (if more than one type is available) #SBATCH --gres=gpu:l40s:1 Practical Examples ================== Example 1: Serial Job --------------------- .. code-block:: bash :caption: serial.sh #!/bin/bash #SBATCH -J serial #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --time=04:00:00 #SBATCH --mem=4G export INPUT="input.dat" export OUTPUT="output.dat" job-nanny ./my_program Example 2: OpenMP Parallel Job ------------------------------ .. code-block:: bash :caption: openmp.sh #!/bin/bash #SBATCH -J openmp #SBATCH --ntasks=1 #SBATCH --cpus-per-task=28 #SBATCH --time=12:00:00 #SBATCH --mem=64G export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK export INPUT="data.dat" export OUTPUT="results/" job-nanny ./program_omp Example 3: MPI Job ------------------ .. code-block:: bash :caption: mpi.sh #!/bin/bash #SBATCH -J mpi #SBATCH --nodes=4 #SBATCH --ntasks-per-node=28 #SBATCH --time=48:00:00 #SBATCH --mem-per-cpu=2G export INPUT="mpi_data/" export OUTPUT="mpi_results/" module load openmpi/4.1.5 job-nanny mpirun -np 112 ./program_mpi Example 4: GPU Job ------------------ .. code-block:: bash :caption: gpu.sh #!/bin/bash #SBATCH -J gpu #SBATCH --partition=gpu #SBATCH --nodes=1 #SBATCH --ntasks=1 #SBATCH --gres=gpu:2 #SBATCH --cpus-per-task=16 #SBATCH --time=12:00:00 #SBATCH --mem=64G export INPUT="gpu_data/" export OUTPUT="gpu_results/" module load cuda/12.0 job-nanny ./program_cuda Example 5: Job Array with Concurrency Limit ------------------------------------------- .. code-block:: bash :caption: array.sh #!/bin/bash #SBATCH -J array #SBATCH --array=1-1000%50 #SBATCH --ntasks=1 #SBATCH --time=01:00:00 #SBATCH --mem=2G export INPUT="data_${SLURM_ARRAY_TASK_ID}.dat" export OUTPUT="result_${SLURM_ARRAY_TASK_ID}.dat" job-nanny ./process Checking Limits and Usage ========================= Running Jobs ------------ .. code-block:: bash # Your jobs squeue -u $USER # All jobs squeue # Only running jobs squeue -t RUNNING # Only pending jobs squeue -t PENDING Resources in Use ---------------- .. code-block:: bash # Cluster usage summary sinfo # Detailed per-node usage sinfo -N -o "%16N %8c %8m %8d %6t" # Available nodes sinfo -t idle Why Is My Job Not Starting? =========================== Common reasons for jobs remaining in **PENDING** state: 1. **Insufficient resources** - All nodes are busy - Solution: Wait, or request fewer resources 2. **Low priority** - According to the :ref:`priority_policy`, your Fair Share is low - Solution: Reduce your usage or wait for the priority reset 3. **Requested resources are infeasible** - Requested more than available (e.g. 100 nodes) - Solution: Review your resource requests 4. **Invalid requested time** - More than 30 days - Solution: Reduce to ≤ 30 days Checking the reason: .. code-block:: bash squeue -u $USER -o "%.18i %.9P %.8j %.8u %.2t %.10M %.6D %R" The **%R** column shows the reason (e.g. "Resources", "Priority"). Optimizing Resource Usage ========================= 1. **Request only what you need** - Estimate time and memory accurately - Use ``sacct`` to see actual usage after the job finishes 2. **Smaller jobs have higher priority** - Prefer multiple small jobs over one large one - Use job arrays for HTC workloads 3. **Avoid submission spikes** - Spread submissions out over time - Use ``--array=1-1000%50`` to limit concurrent tasks 4. **Monitor your Fair Share** .. code-block:: bash sshare -U $USER .. seealso:: - :ref:`priority_policy` - Details about Fair Share - :ref:`running_simulations` - How to submit jobs - :ref:`improving_submission_script` - More script options - :ref:`optimizing_performance` - Optimization strategies