Troubleshooting

This section provides solutions to common problems encountered by GridUnesp users. Problems are organized by category for easy reference.

Tip

Before contacting support, consult this section. Most problems already have a documented solution.

Access Problems

SSH Authentication Error

Symptom: When trying to connect, “Permission denied” or an authentication error appears.

Possible solutions:

  1. Check your credentials

    Connection test with debug
    ssh -vvv username@access.grid.unesp.br

  2. Check the server’s SSH key

    The correct keys for the server access.grid.unesp.br are:

    RSA:     SHA256:X3iCb13fWj7u2Tvp/MCCpn0brfSNS5Ie6ehm6lsvPSQ
ECDSA:   SHA256:WVFokXOLnuH9+2e7xxRU2gp7XJqxwuE6H8bbUMqTCXo
ED25519: SHA256:+HEvFMmo0EA6ipPMJSaj5+Q6IabebRN+nRD0nxdYrKQ

    If the key on your system does not match, remove the old entry:

    ssh-keygen -f "~/.ssh/known_hosts" -R "access.grid.unesp.br"

  3. Check for network blocks

    • Port 22 (SSH) must be open

    • Corporate firewalls may block it

    • Networks with proxies may require special configuration

Connection Refused

Symptom: ssh: connect to host access.grid.unesp.br port 22: Connection refused

Likely cause: Blocked by the Fail2Ban system after multiple failed login attempts or excessive SCP use.

Solution:

  • Wait 15 minutes and try again

  • Do not retry repeatedly during the block (it resets the counter)

  • If the problem persists, send the output of the command below to support:

    ssh -vvv username@access.grid.unesp.br

Slow or Unstable Connection

Symptoms:

  • Commands are slow to respond

  • Connection drops frequently

  • High latency

Solutions:

  1. Use SSH compression:

    ssh -C username@access.grid.unesp.br

  2. Keep the connection alive:

    Configuration in ~/.ssh/config
    Host access.grid.unesp.br
    ServerAliveInterval 60
    ServerAliveCountMax 3

  3. Use screen or tmux for persistent sessions:

    screen -S grid_session
# Inside screen, log in normally
# To detach: Ctrl+A, then D
# To reattach: screen -r grid_session

  4. Check your internet connection (speed, latency, packet loss)

Job Problems

Job Stuck in PENDING State for a Long Time

Symptom: squeue shows your job with state PD (Pending) for hours or days.

Possible causes:

  1. Insufficient available resources

    Check the cluster status:

    sinfo                     # Overview of nodes
squeue -r | wc -l         # Total jobs
squeue -r -t PD | wc -l   # Pending jobs

  2. Requested resources are infeasible

    Check that you have not requested more than what is available:

    • Maximum of 52 CPUs per node (CPU node)

    • Maximum of 88 CPUs in the GPU node

    • Maximum of 30 days of execution

    • Memory compatible with the request

  3. Low priority

    The system uses a Fair Share policy:

    sprio -u $USER             # View priority of your job
squeue -o "%.18i %.9Q %.8j %.8u %.10V %.6D %R" --sort=-p,i --states=PD

  4. Incorrect partition

    Check available partitions:

    sinfo -o "%9P %5a %10l %6D %6t %N"

Solutions:

  • Wait (peak periods may have long queues)

  • Reduce requested resources (fewer CPUs, less time)

  • Submit during lower-demand periods (overnight, weekends)

  • Split into smaller jobs (job array)

Job Fails Immediately

Symptom: Job starts running but fails within seconds (state F or FAILED).

Diagnosis:

  1. Check the output file:

    cat slurm-JOBID.out
tail -50 slurm-JOBID.out

  2. Check the error file (if specified):

    cat slurm-JOBID.err

  3. Check detailed status:

    scontrol show job JOBID

Common problems and solutions:

Error

Cause

Solution

“INPUT is missing”

INPUT variable not defined

Add export INPUT="..."

“OUTPUT is missing”

OUTPUT variable not defined

Add export OUTPUT="..."

“command not found”

Module not loaded

Add module load ...

“Permission denied”

File has no execute permission

chmod +x program

“Segmentation fault”

Code bug or insufficient memory

Increase memory, debug code

“Out of memory”

Program used more memory than requested

Increase --mem or --mem-per-cpu

Job Cancelled (CANCELLED)

Symptom: Job appears as CANCELLED in the history.

Causes:

  1. TIMEOUT: Job exceeded the requested time

    sacct -j JOBID --format=JobID,State,ExitCode,Elapsed,Timelimit

    Solution: Increase #SBATCH -t or optimize the code.

  2. Manual cancellation:

    • You may have cancelled with scancel JOBID

    • Administrators may have cancelled it

  3. Compute node failure:

    • Hardware failure detected

    • Node restarted or under maintenance

    Solution: Resubmit the job (it will be allocated to another node).

  4. Excessive use on the access server:

    • Heavy jobs run on the access node are cancelled by the administrators.

    Solution: Always use sbatch to submit to compute nodes.

Library Error in Job

Symptom: “error while loading shared libraries” or “cannot find -lname”.

Diagnosis:

ldd ./your_executable          # Shows dependencies
module list                    # View loaded modules

Solutions:

  1. Load the correct module:

    module avail partial_name   # Search for the library
module load library_name

  2. Configure LD_LIBRARY_PATH (if installed locally):

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HOME/local/lib

  3. Recompile the program with the correct libraries

Compilation Problems

Symptom: Errors during compilation with make, gcc, etc.

Solutions:

  1. Check available compilers:

    module avail gcc
module avail intel

  2. Load the desired compiler:

    module load gcc/9.3.0
# or
module load intel/2020

  3. For MPI applications, also load the MPI module:

    module load openmpi/4.0.1
# or
module load intel/mpi/2017

  4. Install missing dependencies:

    • Development libraries

    • Required headers

  5. Set appropriate compilation flags:

    CFLAGS="-O2 -march=native" ./configure --prefix=$HOME/local
make
make install

Storage Problems

No Disk Space

Symptom: “No space left on device” or “Disk quota exceeded” error.

Diagnosis:

Check disk usage
df -h  /home               # Total space in /home/
du -sh /home/$USER         # Total used by the user
du -sh /home/$USER/* | sort -hr | head -20   # Largest directories

# Check /store/ (if applicable)
du -sh /store/$USER/* | sort -hr | head -20

Solutions:

  1. Remove unnecessary files:

    rm slurm-*.out                   # Old logs
rm -rf temporary_directories/    # Temporary directories

  2. Compress large files:

    tar -czf results.tar.gz results_directory/
rm -rf results_directory/        # After confirming the tar is OK

  3. Transfer important data to your local computer

  4. Use /store/ for large temporary files during execution:

    export SHARED_FS="true"

Warning

There are no disk quotas on GridUnesp, but the space is shared. Excessive accumulation harms all users.

Files Disappeared

Symptom: Files that were in the directory can no longer be found.

Possible causes:

  1. Automatic /tmp/ cleanup

    • Files in /tmp/ on nodes are deleted after the job

    • Make sure you used job-nanny with OUTPUT defined

  2. Accidental deletion

    • Commands such as rm -rf may have been used incorrectly

    • Check the command history with history | grep rm

  3. Storage failure

    • Hardware failures can cause data corruption/loss

    • GridUnesp does not have automatic backups

Danger

GridUnesp does NOT automatically back up data.

Data lost due to hardware failure cannot be recovered.

Prevention:

  • Keep backups of important data on your local computer

  • Use version control (git) for scripts and code

  • Transfer critical results as soon as the job finishes

Permission Error

Symptom: “Permission denied” when accessing files or directories.

Diagnosis:

ls -la file           # View current permissions
id                    # View your UID/GID

Solutions:

  1. Adjust permissions:

    chmod +r file.dat              # Add read permission
chmod +x script.sh             # Add execute permission
chmod -R g+rw directory/       # Read/write for group

  2. Check ownership:

    chown $USER:$USER file          # Become the owner

  3. To share with colleagues:

    chmod g+rx directory/              # Group can read/execute
chmod o-rwx directory/             # Others have no access

Performance Problems

Very Slow Job

Symptom: Job takes longer than expected to execute.

Causes and solutions:

  1. Inadequate resource usage

    • Check whether the job is using all requested resources:

      sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j JOBID.batch
sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j 23134.batch
sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j JOBID -a
sstat --format=JobID,MaxRSS,MaxVMSize,AveCPU -j 23134 -a

    • Adjust the number of threads/processes in the code

  2. Intensive I/O

    • Many read/write disk operations

    • Consider using /tmp/ (faster) for temporary files

    • Minimize I/O operations (read once, process, write once)

  3. Memory bottleneck

    • Insufficient cache, leading to swapping

    • Increase available memory with #SBATCH --mem=...

  4. Non-optimized code

    • Use optimization flags during compilation (-O2, -O3)

    • Review algorithms (complexity, data structures)

    • Parallelize where possible (OpenMP, MPI)

  5. Contention with other jobs

    • Shared resources (storage, network) may be saturated

    • Try running during lower-demand periods

Out of Memory (OOM)

Symptom: Job is killed with “Out of Memory” or “Killed”.

Diagnosis:

sacct -j JOBID --format=JobID,ReqMem,MaxRSS,State
# ReqMem: requested memory
# MaxRSS: maximum memory actually used

Solutions:

  1. Increase requested memory:

    #SBATCH --mem=16G          # 16 GB per node
# or
#SBATCH --mem-per-cpu=4G   # 4 GB per CPU

  2. Optimize the program’s memory usage:

    • Release unused memory

    • Process in batches instead of loading everything into memory

    • Use more efficient data structures

  3. Reduce the problem (process parts separately)

  4. Use algorithms with a smaller memory footprint

Module Problems

Module Not Found

Symptom: “module: command not found” or “module load name” fails.

Solutions:

  1. Check whether the module environment is initialized:

    which module
module --version

    If not available, the system may have a problem — contact support.

  2. List available modules:

    module avail                 # All modules
module avail partial_name    # Search by partial name

  3. Check whether the module exists with the correct name:

    • Case sensitive (OpenMPI ≠ openmpi)

    • Specific versions (gcc/9.3.0 ≠ gcc)

Module Conflict

Symptom: When loading a module, another is automatically unloaded.

Cause: Incompatible modules (e.g. different MPI versions).

Solutions:

  1. Use separate environments:

    module purge                  # Unload all
module load openmpi/4.0.1     # Load only what is needed

    Warning

    The job-nanny script is recognized when the gridunesp module is loaded.

    When the user logs in to the cluster, this module is loaded automatically.

    If you have used the module purge command, you need to run

    module load gridunesp

    to make job-nanny available again.

    See the job-nanny Script section.

  2. Check dependencies:

    module show module_name       # Shows dependencies and conflicts

  3. Use module swap to switch versions:

    module swap gcc gcc/9.3.0

job-nanny Problems

Required INPUT/OUTPUT

Symptom: Job cancelled with an error about INPUT/OUTPUT.

Cause: The job-nanny script requires the INPUT and OUTPUT environment variables to be defined.

Solution:

Correct example
#!/bin/bash
#SBATCH -J my_job

# If there is more than one input,
# separate the names with whitespace
export INPUT="input_data.dat parameters.txt"
export OUTPUT="output_results.dat logs/"

module load my_software
job-nanny ./my_program

Using /store/ vs /tmp/

Symptom: Slow job or space problems.

Solution: Choose the appropriate location:

Force use of /store/ (for large files)
export SHARED_FS="true"    # Uses /store/
# or
export LARGE_FILES="true"  # Same effect
Force use of /tmp/ (default for 1 node)
export SHARED_FS="false"   # Uses /tmp/ (only for 1 node)

Note

With multiple nodes (-N >1), use of /store/ is automatic and cannot be changed.

Network Problems

Slow Transfer

Symptom: SCP/RSYNC very slow.

Solutions:

  1. Use compression:

    scp -C file.dat user@access.grid.unesp.br:~/

  2. Use RSYNC (more efficient for directories):

    rsync -avz directory/ user@access.grid.unesp.br:~/directory/

  3. Compress before transferring:

    tar -czf data.tar.gz directory/
scp data.tar.gz user@access.grid.unesp.br:~/

  4. Avoid peak hours. Use the night or weekends.

Interrupted Transfer

Symptom: Transfer interrupted mid-way.

Solutions:

  1. Use RSYNC (allows resuming):

    rsync -avz --partial --progress file.dat user@access.grid.unesp.br:~/

  2. Split large files:

    split -b 1G large_file.dat part_   # Split into 1 GB parts
# Transfer the parts
cat part_* > large_file.dat        # Reassemble at destination

  3. Use screen/tmux to prevent the transfer from dying when the terminal is closed

General Problems

Command Not Found

Symptom: “command not found” for common commands.

Solutions:

  1. Check PATH:

    echo $PATH
which command_name

  2. Some common commands may not be in PATH — use the full path:

    /usr/bin/command_name
/bin/command_name

  3. Install locally if it is a specific program. See the Installing Applications section.

Default Shell

Symptom: Prefers another shell (zsh, fish, etc.) but the default is bash.

Solution: Change your default shell:

chsh -s /bin/zsh          # Example for zsh
# Log out and log in again

Note

Not all shells are available. The default is bash.

System Language

Symptom: Messages are in English; prefer Portuguese.

Solution: Set the environment variable:

export LANG=pt_BR.UTF-8
export LANGUAGE=pt_BR

To make it permanent, add to ~/.bashrc.

When to Contact Support

If the problem persists after trying the solutions above:

Prepare the following information:

  1. Clear description of the problem

  2. Your username

  3. Job IDs (if applicable)

  4. Full path of scripts and files

  5. Contents of log files (slurm-JOBID.out)

  6. Commands executed and their output

  7. Solutions already attempted

Send to: support.ncc@unesp.br

Tip

The more complete your report, the faster and more accurate the response will be.

See also