.. _troubleshooting:

===============
Troubleshooting
===============

.. contents:: In this section:
   :local:
   :depth: 2

This section provides solutions to common problems encountered by GridUnesp users. Problems are organized by category for easy reference.

.. tip::
   Before contacting support, consult this section. Most problems already have a documented solution.

Access Problems
===============

SSH Authentication Error
------------------------

**Symptom:** When trying to connect, "Permission denied" or an authentication error appears.

**Possible solutions:**

1. **Check your credentials**

   .. code-block:: bash
      :caption: Connection test with debug

      ssh -vvv username@access.grid.unesp.br

   - Make sure you are using the **correct username**
   - Check that the **password is correct** (no Caps Lock)
   - If you forgot your password, reset it at: https://www.ncc.unesp.br/password-new/


2. **Check the server's SSH key**

   The correct keys for the server access.grid.unesp.br are:

   .. code-block:: text

      RSA:     SHA256:X3iCb13fWj7u2Tvp/MCCpn0brfSNS5Ie6ehm6lsvPSQ
      ECDSA:   SHA256:WVFokXOLnuH9+2e7xxRU2gp7XJqxwuE6H8bbUMqTCXo
      ED25519: SHA256:+HEvFMmo0EA6ipPMJSaj5+Q6IabebRN+nRD0nxdYrKQ

   If the key on your system does not match, remove the old entry:

   .. code-block:: bash

      ssh-keygen -f "~/.ssh/known_hosts" -R "access.grid.unesp.br"

3. **Check for network blocks**

   - Port 22 (SSH) must be open
   - Corporate firewalls may block it
   - Networks with proxies may require special configuration

Connection Refused
------------------

**Symptom:** ``ssh: connect to host access.grid.unesp.br port 22: Connection refused``

**Likely cause:** Blocked by the **Fail2Ban** system after multiple failed login attempts or excessive SCP use.

**Solution:**

- Wait **15 minutes** and try again
- Do not retry repeatedly during the block (it resets the counter)
- If the problem persists, send the output of the command below to support:

  .. code-block:: bash

     ssh -vvv username@access.grid.unesp.br

Slow or Unstable Connection
---------------------------

**Symptoms:**

- Commands are slow to respond
- Connection drops frequently
- High latency

**Solutions:**

1. **Use SSH compression:**

   .. code-block:: bash

      ssh -C username@access.grid.unesp.br

2. **Keep the connection alive:**

   .. code-block:: bash
      :caption: Configuration in ~/.ssh/config

      Host access.grid.unesp.br
          ServerAliveInterval 60
          ServerAliveCountMax 3

3. **Use screen or tmux** for persistent sessions:

   .. code-block:: bash

      screen -S grid_session
      # Inside screen, log in normally
      # To detach: Ctrl+A, then D
      # To reattach: screen -r grid_session

4. **Check your internet connection** (speed, latency, packet loss)

Job Problems
============

Job Stuck in PENDING State for a Long Time
------------------------------------------

**Symptom:** ``squeue`` shows your job with state **PD** (Pending) for hours or days.

**Possible causes:**

1. **Insufficient available resources**

   Check the cluster status:

   .. code-block:: bash

      sinfo                     # Overview of nodes
      squeue -r | wc -l         # Total jobs
      squeue -r -t PD | wc -l   # Pending jobs

2. **Requested resources are infeasible**

   Check that you have not requested more than what is available:

   - Maximum of 52 CPUs per node (CPU node)
   - Maximum of 88 CPUs in the GPU node
   - Maximum of 30 days of execution
   - Memory compatible with the request

3. **Low priority**

   The system uses a Fair Share policy:

   .. code-block:: bash

      sprio -u $USER             # View priority of your job
      squeue -o "%.18i %.9Q %.8j %.8u %.10V %.6D %R" --sort=-p,i --states=PD

4. **Incorrect partition**

   Check available partitions:

   .. code-block:: bash

      sinfo -o "%9P %5a %10l %6D %6t %N"

**Solutions:**

- Wait (peak periods may have long queues)
- Reduce requested resources (fewer CPUs, less time)
- Submit during lower-demand periods (overnight, weekends)
- Split into smaller jobs (job array)

Job Fails Immediately
---------------------

**Symptom:** Job starts running but fails within seconds (state **F** or **FAILED**).

**Diagnosis:**

1. **Check the output file:**

   .. code-block:: bash

      cat slurm-JOBID.out
      tail -50 slurm-JOBID.out

2. **Check the error file (if specified):**

   .. code-block:: bash

      cat slurm-JOBID.err

3. **Check detailed status:**

   .. code-block:: bash

      scontrol show job JOBID

**Common problems and solutions:**

==================== =============================================== ======================================
**Error**            **Cause**                                       **Solution**
-------------------- ----------------------------------------------- --------------------------------------
"INPUT is missing"   INPUT variable not defined                      Add ``export INPUT="..."``
"OUTPUT is missing"  OUTPUT variable not defined                     Add ``export OUTPUT="..."``
"command not found"  Module not loaded                               Add ``module load ...``
"Permission denied"  File has no execute permission                  ``chmod +x program``
"Segmentation fault" Code bug or insufficient memory                 Increase memory, debug code
"Out of memory"      Program used more memory than requested         Increase ``--mem`` or ``--mem-per-cpu``
==================== =============================================== ======================================

Job Cancelled (CANCELLED)
-------------------------

**Symptom:** Job appears as **CANCELLED** in the history.

**Causes:**

1. **TIMEOUT:** Job exceeded the requested time

   .. code-block:: bash

      sacct -j JOBID --format=JobID,State,ExitCode,Elapsed,Timelimit

   **Solution:** Increase ``#SBATCH -t`` or optimize the code.

2. **Manual cancellation:**

   - You may have cancelled with ``scancel JOBID``
   - Administrators may have cancelled it

3. **Compute node failure:**

   - Hardware failure detected
   - Node restarted or under maintenance

   **Solution:** Resubmit the job (it will be allocated to another node).

4. **Excessive use on the access server:**

   - Heavy jobs run on the access node are cancelled by the administrators.

   **Solution:** Always use ``sbatch`` to submit to compute nodes.

Library Error in Job
--------------------

**Symptom:** "error while loading shared libraries" or "cannot find -lname".

**Diagnosis:**

.. code-block:: bash

   ldd ./your_executable          # Shows dependencies
   module list                    # View loaded modules

**Solutions:**

1. **Load the correct module:**

   .. code-block:: bash

      module avail partial_name   # Search for the library
      module load library_name

2. **Configure LD_LIBRARY_PATH** (if installed locally):

   .. code-block:: bash

      export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/local/lib

3. **Recompile the program** with the correct libraries

Compilation Problems
--------------------

**Symptom:** Errors during compilation with ``make``, ``gcc``, etc.

**Solutions:**

1. **Check available compilers:**

   .. code-block:: bash

      module avail gcc
      module avail intel

2. **Load the desired compiler:**

   .. code-block:: bash

      module load gcc/9.3.0
      # or
      module load intel/2020

3. **For MPI applications, also load the MPI module:**

   .. code-block:: bash

      module load openmpi/4.0.1
      # or
      module load intel/mpi/2017

4. **Install missing dependencies:**

   - Development libraries
   - Required headers

5. **Set appropriate compilation flags:**

   .. code-block:: bash

      CFLAGS="-O2 -march=native" ./configure --prefix=$HOME/local
      make
      make install

Storage Problems
================

No Disk Space
-------------

**Symptom:** "No space left on device" or "Disk quota exceeded" error.

**Diagnosis:**

.. code-block:: bash
   :caption: Check disk usage

   df -h  /home               # Total space in /home/
   du -sh /home/$USER         # Total used by the user
   du -sh /home/$USER/* | sort -hr | head -20   # Largest directories

   # Check /store/ (if applicable)
   du -sh /store/$USER/* | sort -hr | head -20

**Solutions:**

1. **Remove unnecessary files:**

   .. code-block:: bash

      rm slurm-*.out                   # Old logs
      rm -rf temporary_directories/    # Temporary directories

2. **Compress large files:**

   .. code-block:: bash

      tar -czf results.tar.gz results_directory/
      rm -rf results_directory/        # After confirming the tar is OK

3. **Transfer important data** to your local computer

4. **Use /store/ for large temporary files** during execution:

   .. code-block:: bash

      export SHARED_FS="true"

.. warning::
   **There are no disk quotas** on GridUnesp, but the space is shared. Excessive accumulation harms all users.

Files Disappeared
-----------------

**Symptom:** Files that were in the directory can no longer be found.

**Possible causes:**

1. **Automatic /tmp/ cleanup**

   - Files in ``/tmp/`` on nodes are deleted after the job
   - Make sure you used ``job-nanny`` with OUTPUT defined

2. **Accidental deletion**

   - Commands such as ``rm -rf`` may have been used incorrectly
   - Check the command history with ``history | grep rm``

3. **Storage failure**

   - Hardware failures can cause data corruption/loss
   - GridUnesp **does not have automatic backups**

.. danger::
   **GridUnesp does NOT automatically back up data.**

   Data lost due to hardware failure **cannot be recovered**.

**Prevention:**

- Keep backups of important data on your local computer
- Use version control (git) for scripts and code
- Transfer critical results as soon as the job finishes

Permission Error
----------------

**Symptom:** "Permission denied" when accessing files or directories.

**Diagnosis:**

.. code-block:: bash

   ls -la file           # View current permissions
   id                    # View your UID/GID

**Solutions:**

1. **Adjust permissions:**

   .. code-block:: bash

      chmod +r file.dat              # Add read permission
      chmod +x script.sh             # Add execute permission
      chmod -R g+rw directory/       # Read/write for group

2. **Check ownership:**

   .. code-block:: bash

      chown $USER:$USER file          # Become the owner

3. **To share with colleagues:**

   .. code-block:: bash

      chmod g+rx directory/              # Group can read/execute
      chmod o-rwx directory/             # Others have no access

Performance Problems
====================

Very Slow Job
-------------

**Symptom:** Job takes longer than expected to execute.

**Causes and solutions:**

1. **Inadequate resource usage**

   - Check whether the job is using all requested resources:

     .. code-block:: bash

        sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j JOBID.batch
        sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j 23134.batch
        sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j JOBID -a
        sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j 23134 -a

   - Adjust the number of threads/processes in the code

2. **Intensive I/O**

   - Many read/write disk operations
   - Consider using ``/tmp/`` (faster) for temporary files
   - Minimize I/O operations (read once, process, write once)

3. **Memory bottleneck**

   - Insufficient cache, leading to swapping
   - Increase available memory with ``#SBATCH --mem=...``

4. **Non-optimized code**

   - Use optimization flags during compilation (``-O2``, ``-O3``)
   - Review algorithms (complexity, data structures)
   - Parallelize where possible (OpenMP, MPI)

5. **Contention with other jobs**

   - Shared resources (storage, network) may be saturated
   - Try running during lower-demand periods

Out of Memory (OOM)
-------------------

**Symptom:** Job is killed with "Out of Memory" or "Killed".

**Diagnosis:**

.. code-block:: bash

   sacct -j JOBID --format=JobID,ReqMem,MaxRSS,State
   # ReqMem: requested memory
   # MaxRSS: maximum memory actually used

**Solutions:**

1. **Increase requested memory:**

   .. code-block:: bash

      #SBATCH --mem=16G          # 16 GB per node
      # or
      #SBATCH --mem-per-cpu=4G   # 4 GB per CPU

2. **Optimize the program's memory usage:**

   - Release unused memory
   - Process in batches instead of loading everything into memory
   - Use more efficient data structures

3. **Reduce the problem** (process parts separately)

4. **Use algorithms with a smaller memory footprint**

Module Problems
===============

Module Not Found
----------------

**Symptom:** "module: command not found" or "module load name" fails.

**Solutions:**

1. **Check whether the module environment is initialized:**

   .. code-block:: bash

      which module
      module --version

   If not available, the system may have a problem — contact support.

2. **List available modules:**

   .. code-block:: bash

      module avail                 # All modules
      module avail partial_name    # Search by partial name

3. **Check whether the module exists with the correct name:**

   - Case sensitive (OpenMPI ≠ openmpi)
   - Specific versions (gcc/9.3.0 ≠ gcc)

Module Conflict
---------------

**Symptom:** When loading a module, another is automatically unloaded.

**Cause:** Incompatible modules (e.g. different MPI versions).

**Solutions:**

1. **Use separate environments:**

   .. code-block:: bash

      module purge                  # Unload all
      module load openmpi/4.0.1     # Load only what is needed

   .. warning::
      The ``job-nanny`` script is recognized when the **gridunesp** module is loaded.

      When the user logs in to the cluster, this module is loaded automatically.

      If you have used the ``module purge`` command, you need to run

      .. code-block:: bash

         module load gridunesp

      to make ``job-nanny`` available again.

      See the :ref:`job_nanny_script` section.

2. **Check dependencies:**

   .. code-block:: bash

      module show module_name       # Shows dependencies and conflicts

3. **Use module swap** to switch versions:

   .. code-block:: bash

      module swap gcc gcc/9.3.0

job-nanny Problems
==================

Required INPUT/OUTPUT
---------------------

**Symptom:** Job cancelled with an error about INPUT/OUTPUT.

**Cause:** The ``job-nanny`` script requires the INPUT and OUTPUT environment variables to be defined.

**Solution:**

.. code-block:: bash
   :caption: Correct example

   #!/bin/bash
   #SBATCH -J my_job

   # If there is more than one input,
   # separate the names with whitespace
   export INPUT="input_data.dat parameters.txt"
   export OUTPUT="output_results.dat logs/"

   module load my_software
   job-nanny ./my_program

Using /store/ vs /tmp/
----------------------

**Symptom:** Slow job or space problems.

**Solution:** Choose the appropriate location:

.. code-block:: bash
   :caption: Force use of /store/ (for large files)

   export SHARED_FS="true"    # Uses /store/
   # or
   export LARGE_FILES="true"  # Same effect

.. code-block:: bash
   :caption: Force use of /tmp/ (default for 1 node)

   export SHARED_FS="false"   # Uses /tmp/ (only for 1 node)

.. note::
   With multiple nodes (``-N >1``), use of ``/store/`` is automatic and cannot be changed.

Network Problems
================

Slow Transfer
-------------

**Symptom:** SCP/RSYNC very slow.

**Solutions:**

1. **Use compression:**

   .. code-block:: bash

      scp -C file.dat user@access.grid.unesp.br:~/

2. **Use RSYNC (more efficient for directories):**

   .. code-block:: bash

      rsync -avz directory/ user@access.grid.unesp.br:~/directory/

3. **Compress before transferring:**

   .. code-block:: bash

      tar -czf data.tar.gz directory/
      scp data.tar.gz user@access.grid.unesp.br:~/

4. **Avoid peak hours.** Use the night or weekends.

Interrupted Transfer
--------------------

**Symptom:** Transfer interrupted mid-way.

**Solutions:**

1. **Use RSYNC (allows resuming):**

   .. code-block:: bash

      rsync -avz --partial --progress file.dat user@access.grid.unesp.br:~/

2. **Split large files:**

   .. code-block:: bash

      split -b 1G large_file.dat part_   # Split into 1 GB parts
      # Transfer the parts
      cat part_* > large_file.dat        # Reassemble at destination

3. **Use screen/tmux** to prevent the transfer from dying when the terminal is closed

General Problems
================

Command Not Found
-----------------

**Symptom:** "command not found" for common commands.

**Solutions:**

1. **Check PATH:**

   .. code-block:: bash

      echo $PATH
      which command_name

2. **Some common commands may not be in PATH** — use the full path:

   .. code-block:: bash

      /usr/bin/command_name
      /bin/command_name

3. **Install locally** if it is a specific program. See the :ref:`installing_applications` section.

Default Shell
-------------

**Symptom:** Prefers another shell (zsh, fish, etc.) but the default is bash.

**Solution:** Change your default shell:

.. code-block:: bash

   chsh -s /bin/zsh          # Example for zsh
   # Log out and log in again

.. note::
   Not all shells are available. The default is bash.

System Language
---------------

**Symptom:** Messages are in English; prefer Portuguese.

**Solution:** Set the environment variable:

.. code-block:: bash

   export LANG=pt_BR.UTF-8
   export LANGUAGE=pt_BR

To make it permanent, add to ``~/.bashrc``.

When to Contact Support
=======================

If the problem persists after trying the solutions above:

**Prepare the following information:**

1. **Clear description of the problem**
2. **Your username**
3. **Job IDs** (if applicable)
4. **Full path of scripts and files**
5. **Contents of log files** (slurm-JOBID.out)
6. **Commands executed** and their output
7. **Solutions already attempted**

**Send to:** ``support.ncc@unesp.br``

.. tip::
   The more complete your report, the faster and more accurate the response will be.

.. seealso::

   - :ref:`faq` - Common questions
   - :ref:`best_practices` - Usage recommendations
   - :ref:`contact` - Support channels