Complete Storage Guide

In this section:

Understanding where and how to store your data on GridUnesp is crucial for the performance of your jobs and for the safety of your data.

Danger

CRITICAL WARNING:

❌ NO BACKUP of any data on GridUnesp
❌ NO DISK QUOTAS (use responsibly)
⚠️ You are 100% responsible for your data
⚠️ Hardware failures, human errors or technical problems can cause permanent data loss

If the data is important, back it up somewhere else!

Storage Systems 

GridUnesp has three main storage systems:

Comparison of systems
System	Capacity	Speed	Persistence	Recommended use
`/home/`	40 TB	Medium (NFS)	Permanent	Scripts, code, small data files
`/tmp/`	~120 GB/node	⚡⚡⚡ Very fast (local)	Temporary (only during the job)	Intensive I/O, temporary files
`/store/`	7 TB	⚡ Medium (NFS)	Permanent	Large files (>100 GB), shared data

/home/ - Personal Directory 

Your /home/your-username directory is your permanent personal space on the cluster.

Characteristics 

Access: Available on all nodes (access, compute, GPU)
Speed: Medium (accessed via NFS network)
Backup: NO (data is NOT copied)
Quota: NO (use responsibly)
Persistence: Permanent (data remains until you delete it)

What to Store in /home/

✅ Recommended:

Submission scripts (.sh)
Source code
Small executables
Configuration files
Small data files (<1 GB)
Important final results

❌ Not recommended:

Temporary job files
Bulky intermediate data
Very large files (>10 GB)
Data that will be processed intensively

Suggested Organization:

/home/your-username/
├── projects/
│   ├── project1/
│   │   ├── code/
│   │   ├── scripts/
│   │   └── final_results/
│   └── project2/
├── software/
│   ├── program1/
│   └── program2/
└── bin/              # Personal executables

/tmp/ - Temporary Local Storage 

Each compute node has its own local /tmp/ directory.

Characteristics 

Access: Only on the node where the job is running
Speed: ⚡⚡⚡ Very fast (local SSD/HD)
Capacity: Limited (~120 GB per node)
Persistence: TEMPORARY - files are removed at the end of the job
Sharing: Not shared between nodes

Warning

IMPORTANT NOTE ABOUT /tmp/:

Files in /tmp/ may be deleted without notice
When the job ends, data in /tmp/ is lost
NEVER use /tmp/ for unique or important data
ALWAYS copy important results from /tmp/ to /home/ or /store/

When to Use /tmp/

✅ Ideal for:

Temporary files during execution
Discardable intermediate data
Intensive I/O (frequent reads/writes)
Temporary checkpoints (if they can be recreated)

❌ Never use for:

Final results
Unique, non-recreatable data
Storage between jobs

Using /tmp/ Correctly 

using_tmp.sh

#!/bin/bash
#SBATCH -J tmp_example
#SBATCH -N 1
#SBATCH -n 28
#SBATCH -t 06:00:00

export INPUT="input_data.dat"
export OUTPUT="final_result.dat"
# job-nanny will use /tmp/ automatically (1 node, no flags)

job-nanny ./processor input_data.dat

/store/ - Shared Storage 

/store/ is a shared filesystem visible to all nodes.

Characteristics 

Access: Visible on all nodes
Speed: Medium (accessed via NFS network)
Capacity: 7 TB (shared)
Persistence: Permanent
Sharing: Visible to all nodes

When to Use /store/

✅ Ideal for:

Multi-node jobs (MPI)
Very large input/output files (>100 GB)
Data shared among users
Reference datasets

⚠️ Use with care:

Not optimized for intensive I/O
Can become slow with many simultaneous accesses

Using /store/

Automatically (multi-node jobs):

job_multinode.sh

#!/bin/bash
#SBATCH -J mpi_job
#SBATCH -N 4
#SBATCH --ntasks-per-node=28
#SBATCH -t 48:00:00

export INPUT="large_data/"
export OUTPUT="mpi_results/"
# With -N 4, job-nanny will use /store/ automatically

module load openmpi/4.1.5
job-nanny mpirun -np $SLURM_NTASKS ./program_mpi

Forcing /store/ for a single-node job:

job_large_1node.sh

#!/bin/bash
#SBATCH -J big_job
#SBATCH -N 1
#SBATCH -n 28
#SBATCH -t 72:00:00

export INPUT="dataset_500GB.bin"
export OUTPUT="results/"
export LARGE_FILES="true"      # Forces use of /store/

job-nanny srun ./processor dataset_500GB.bin

Managing Disk Space 

Checking Disk Usage 

# Total usage of your /home/
du -sh /home/$USER

# Detailed usage by subdirectory
du -h --max-depth=1 /home/$USER | sort -hr

# Find large files (>1 GB)
find /home/$USER -type f -size +1G -exec ls -lh {} \;

# See free space on the system
df -h /home
df -h /store

Example output:

$ du -sh /home/john
45G     /home/john

$ du -h --max-depth=1 /home/john | sort -hr
28G     /home/john/project2
15G     /home/john/project1
2.1G    /home/john/software
512M    /home/john/scripts
45G     /home/john

Cleaning Up Files 

# Remove old log files (>90 days)
find /home/$USER -name "*.log" -mtime +90 -delete
find /home/$USER -name "slurm-*.out" -mtime +30 -delete

# Remove temporary directories
rm -rf /home/$USER/tmp/
rm -rf /home/$USER/old_results/

# Compress old data (before removing)
tar -czf old_project.tar.gz old_project/
rm -rf old_project/

# Clean Conda cache (if used)
conda clean --all

Automatic Cleanup Policy 

/tmp/:

Files in /tmp/ are automatically removed at the end of the job
They may be removed earlier if the node needs space
There is NO GUARANTEE of how long files will remain in /tmp/

/home/ and /store/:

Currently no automatic cleanup is performed
If the job completes successfully, job-nanny automatically removes files/directories from /store
However, excessive accumulation can lead to:
- System slowdowns
- Job failures due to lack of space
- Cleanup requests from the team

Recommendations:

Keep only what is needed for current/future processing
Transfer important results to your local computer
Regularly clean up temporary files and old results
Use /store/ for large data, but keep it organized

Backup Strategies 

Given the absence of backup on GridUnesp, you must implement your own strategies.

Option 1: Manual Backup with rsync 

# From your local computer, back up GridUnesp
rsync -avz --progress your-username@access.grid.unesp.br:/home/your-username/ ~/backup-gridunesp/

# Automate with cron (Linux/Mac)
# Add to crontab (crontab -e):
0 2 * * 0 rsync -avz your-username@access.grid.unesp.br:/home/your-username/ ~/backup-gridunesp/

Option 2: Git for Code 

# On GridUnesp, inside your project
git init
git add .
git commit -m "Initial commit"

# Connect to GitHub/GitLab (create repository first)
git remote add origin https://github.com/your-username/your-project.git
git push -u origin main

Advantages:

Code versioning
Automatic cloud backup
Easy collaboration

Option 3: Cloud Storage (rclone)

# Configure rclone (once)
# Follow the instructions to connect to Google Drive, Dropbox, etc.
rclone config

# Perform backup
rclone sync /home/$USER/important-project remote:backup-gridunesp/

# Schedule with cron
0 3 * * * rclone sync /home/$USER/important-results remote:backup/

Option 4: Periodic Tarball 

backup_script.sh

#!/bin/bash
# Script to create a compressed backup

DATE=$(date +%Y-%m-%d)
BACKUP_DIR="/home/$USER/backups"
mkdir -p $BACKUP_DIR

# List of important directories
DIRS=(
    "/home/$USER/project1"
    "/home/$USER/project2"
    "/home/$USER/scripts"
)

# Create tar.gz file
tar -czf $BACKUP_DIR/backup-$DATE.tar.gz "${DIRS[@]}"

# Copy to a safe location (example: institutional server)
scp $BACKUP_DIR/backup-$DATE.tar.gz user@safe-server:/backups/

# Keep only the last 5 backups
ls -t $BACKUP_DIR/backup-*.tar.gz | tail -n +6 | xargs -r rm

Best Practices 

3-2-1 Rule 

3 copies of the data
On 2 different media
1 copy off-site

Backup Checklist 

[ ] Source code in Git (GitHub/GitLab)
[ ] Important results copied to local computer
[ ] Scripts and configurations versioned
[ ] Raw data (if not recreatable) has an external copy
[ ] Backup tested periodically (try restoring!)

I/O Optimization 

For better read/write performance:

Minimize the number of files

# Bad: 10000 small files
for i in {1..10000}; do
    echo "$i" >> file_$i.txt
done

# Good: 1 file
for i in {1..10000}; do
    echo "$i"
done > all_data.txt

Use efficient binary formats
- HDF5 for multidimensional scientific data
- NetCDF for geospatial/climate data
- Parquet for large tabular data
- NPY/NPZ for NumPy arrays

Compress inactive data

# Compress
tar -czf data-2023.tar.gz data-2023/
rm -rf data-2023/

# When needed (slower, but saves space)
tar -xzf data-2023.tar.gz

Troubleshooting 

“No space left on device”

# Check usage
df -h /home
du -sh /home/$USER | sort -hr | head -20

# Clean up large/unnecessary files
find /home/$USER -name "core.*" -delete
find /home/$USER -name "*.tmp" -delete
find /home/$USER -name "slurm-*.out" -mtime +30 -delete

“Permission denied” when accessing /store/

# Check permissions
ls -la /store/$USER

# Fix if necessary
### Full access for the owner (read, write and execute)
### and read and execute permission for group and other users
chmod 755 /store/$USER
### Allows owner to read and write,
### and other users to read only
chmod 644 /store/$USER/*

Files disappeared from /tmp/

This is normal and expected. /tmp/ is temporary!
Solution: * Always copy results to /home/ or /store/ * job-nanny copies them automatically at the end of successfully completed jobs

Slow I/O performance

Check whether you are using /tmp/ for intensive I/O
Reduce the number of small files
Use binary formats instead of text
Avoid listing directories with many files (ls)

Frequently Asked Questions 

Q: Can I increase my disk quota?

A: There are no quotas! But use the space responsibly. It is shared among all users.

Q: How long does data stay in /store/?

A: Permanently, until you delete it or hardware fails. Back it up!

Q: Why was my job data in /store/ deleted?

A: job-nanny automatically deletes data after jobs that finished successfully.

Q: Can I recover accidentally deleted data?

A: No. GridUnesp has no recovery system. Use backups.

Q: How do I share data with colleagues?

A: Use /store/ and adjust group permissions, or use /home/ with appropriate permissions.

Q: Can I mount /home/ on my local computer?

A: Not directly. Use scp, rsync, or sshfs for remote access.

Q: What happens if /store/ fills up?

A: Jobs may fail and the system may slow down. Clean up unnecessary data.