.. _storage_guide:

======================
Complete Storage Guide
======================

.. contents:: In this section:
   :local:
   :depth: 2

Understanding where and how to store your data on GridUnesp is crucial for the performance of your jobs and for the safety of your data.

.. danger::
   **CRITICAL WARNING:**

   - ❌ **NO BACKUP** of any data on GridUnesp
   - ❌ **NO DISK QUOTAS** (use responsibly)
   - ⚠️ You are **100% responsible** for your data
   - ⚠️ Hardware failures, human errors or technical problems can cause **permanent data loss**

   **If the data is important, back it up somewhere else!**

Storage Systems
===============

GridUnesp has three main storage systems:

.. list-table:: Comparison of systems
   :header-rows: 1
   :widths: 15 20 20 20 25

   * - System
     - Capacity
     - Speed
     - Persistence
     - Recommended use
   * - ``/home/``
     - 40 TB
     - Medium (NFS)
     - Permanent
     - Scripts, code, small data files
   * - ``/tmp/``
     - ~120 GB/node
     - ⚡⚡⚡ Very fast (local)
     - Temporary (only during the job)
     - Intensive I/O, temporary files
   * - ``/store/``
     - 7 TB
     - ⚡ Medium (NFS)
     - Permanent
     - Large files (>100 GB), shared data

.. _home_partition:

/home/ - Personal Directory
===========================

Your ``/home/your-username`` directory is your permanent personal space on the cluster.

Characteristics
---------------

- **Access:** Available on all nodes (access, compute, GPU)
- **Speed:** Medium (accessed via NFS network)
- **Backup:** **NO** (data is NOT copied)
- **Quota:** **NO** (use responsibly)
- **Persistence:** Permanent (data remains until you delete it)

What to Store in /home/
-----------------------

✅ **Recommended:**

- Submission scripts (``.sh``)
- Source code
- Small executables
- Configuration files
- Small data files (<1 GB)
- Important final results

❌ **Not recommended:**

- Temporary job files
- Bulky intermediate data
- Very large files (>10 GB)
- Data that will be processed intensively

Suggested Organization:
-----------------------

.. code-block:: text

   /home/your-username/
   ├── projects/
   │   ├── project1/
   │   │   ├── code/
   │   │   ├── scripts/
   │   │   └── final_results/
   │   └── project2/
   ├── software/
   │   ├── program1/
   │   └── program2/
   └── bin/              # Personal executables

.. _tmp_partition_detailed:

/tmp/ - Temporary Local Storage
===============================

Each compute node has its own local ``/tmp/`` directory.

Characteristics
---------------

- **Access:** Only on the node where the job is running
- **Speed:** ⚡⚡⚡ Very fast (local SSD/HD)
- **Capacity:** Limited (~120 GB per node)
- **Persistence:** **TEMPORARY** - files are removed at the end of the job
- **Sharing:** Not shared between nodes

.. warning::
   **IMPORTANT NOTE ABOUT /tmp/**:

   - Files in /tmp/ may be deleted **without notice**
   - When the job ends, data in /tmp/ is lost
   - **NEVER** use /tmp/ for unique or important data
   - **ALWAYS** copy important results from /tmp/ to /home/ or /store/

When to Use /tmp/
-----------------

✅ **Ideal for:**

- Temporary files during execution
- Discardable intermediate data
- Intensive I/O (frequent reads/writes)
- Temporary checkpoints (if they can be recreated)

❌ **Never use for:**

- Final results
- Unique, non-recreatable data
- Storage between jobs

Using /tmp/ Correctly
---------------------

.. code-block:: bash
   :caption: using_tmp.sh

   #!/bin/bash
   #SBATCH -J tmp_example
   #SBATCH -N 1
   #SBATCH -n 28
   #SBATCH -t 06:00:00

   export INPUT="input_data.dat"
   export OUTPUT="final_result.dat"
   # job-nanny will use /tmp/ automatically (1 node, no flags)

   job-nanny ./processor input_data.dat

.. _store_partition_detailed:

/store/ - Shared Storage
========================

``/store/`` is a shared filesystem visible to all nodes.

Characteristics
---------------

- **Access:** Visible on all nodes
- **Speed:** Medium (accessed via NFS network)
- **Capacity:** 7 TB (shared)
- **Persistence:** Permanent
- **Sharing:** Visible to all nodes

When to Use /store/
-------------------

✅ **Ideal for:**

- Multi-node jobs (MPI)
- Very large input/output files (>100 GB)
- Data shared among users
- Reference datasets

⚠️ **Use with care:**

- Not optimized for intensive I/O
- Can become slow with many simultaneous accesses

Using /store/
-------------

**Automatically (multi-node jobs):**

.. code-block:: bash
   :caption: job_multinode.sh

   #!/bin/bash
   #SBATCH -J mpi_job
   #SBATCH -N 4
   #SBATCH --ntasks-per-node=28
   #SBATCH -t 48:00:00

   export INPUT="large_data/"
   export OUTPUT="mpi_results/"
   # With -N 4, job-nanny will use /store/ automatically

   module load openmpi/4.1.5
   job-nanny mpirun -np $SLURM_NTASKS ./program_mpi

**Forcing /store/ for a single-node job:**

.. code-block:: bash
   :caption: job_large_1node.sh

   #!/bin/bash
   #SBATCH -J big_job
   #SBATCH -N 1
   #SBATCH -n 28
   #SBATCH -t 72:00:00

   export INPUT="dataset_500GB.bin"
   export OUTPUT="results/"
   export LARGE_FILES="true"      # Forces use of /store/

   job-nanny srun ./processor dataset_500GB.bin

Managing Disk Space
===================

Checking Disk Usage
-------------------

.. code-block:: bash

   # Total usage of your /home/
   du -sh /home/$USER

   # Detailed usage by subdirectory
   du -h --max-depth=1 /home/$USER | sort -hr

   # Find large files (>1 GB)
   find /home/$USER -type f -size +1G -exec ls -lh {} \;

   # See free space on the system
   df -h /home
   df -h /store

**Example output:**

.. code-block:: text

   $ du -sh /home/john
   45G     /home/john

   $ du -h --max-depth=1 /home/john | sort -hr
   28G     /home/john/project2
   15G     /home/john/project1
   2.1G    /home/john/software
   512M    /home/john/scripts
   45G     /home/john

Cleaning Up Files
-----------------

.. code-block:: bash

   # Remove old log files (>90 days)
   find /home/$USER -name "*.log" -mtime +90 -delete
   find /home/$USER -name "slurm-*.out" -mtime +30 -delete

   # Remove temporary directories
   rm -rf /home/$USER/tmp/
   rm -rf /home/$USER/old_results/

   # Compress old data (before removing)
   tar -czf old_project.tar.gz old_project/
   rm -rf old_project/

   # Clean Conda cache (if used)
   conda clean --all

Sharing Data with Other Users
=============================

To share data with project colleagues:

.. code-block:: bash

   # Create a shared directory in /store/ (recommended for shared data)
   mkdir -p /store/shared_project
   chmod 750 /store/shared_project

   # Copy data
   cp -r your_data /store/shared_project/

   # Adjust permissions (group can read/execute)
   chmod -R 750 /store/shared_project
   chgrp -R group_name /store/shared_project

   # Verify permissions
   ls -la /store/shared_project

For smaller files in /home/:

.. code-block:: bash

   # Grant read permission to the group
   chmod g+r shared_file.dat
   chmod g+rx shared_directory/

   # Remove permissions for others (security)
   chmod o-rwx shared_directory/

Automatic Cleanup Policy
========================

**/tmp/**:

- Files in ``/tmp/`` are **automatically removed** at the end of the job
- They may be removed earlier if the node needs space
- There is **NO GUARANTEE** of how long files will remain in /tmp/

**/home/ and /store/**:

- Currently **no automatic cleanup** is performed
- If the job completes successfully, ``job-nanny`` automatically removes files/directories from ``/store``
- However, excessive accumulation can lead to:

  - System slowdowns
  - Job failures due to lack of space
  - Cleanup requests from the team

**Recommendations:**

- Keep only what is needed for current/future processing
- Transfer important results to your local computer
- Regularly clean up temporary files and old results
- Use ``/store/`` for large data, but keep it organized

Backup Strategies
=================

Given the absence of backup on GridUnesp, you must implement your own strategies.

Option 1: Manual Backup with rsync
----------------------------------

.. code-block:: bash

   # From your local computer, back up GridUnesp
   rsync -avz --progress your-username@access.grid.unesp.br:/home/your-username/ ~/backup-gridunesp/

   # Automate with cron (Linux/Mac)
   # Add to crontab (crontab -e):
   0 2 * * 0 rsync -avz your-username@access.grid.unesp.br:/home/your-username/ ~/backup-gridunesp/

Option 2: Git for Code
----------------------

.. code-block:: bash

   # On GridUnesp, inside your project
   git init
   git add .
   git commit -m "Initial commit"

   # Connect to GitHub/GitLab (create repository first)
   git remote add origin https://github.com/your-username/your-project.git
   git push -u origin main

**Advantages:**

- Code versioning
- Automatic cloud backup
- Easy collaboration

Option 3: Cloud Storage (rclone)
--------------------------------

.. code-block:: bash

   # Configure rclone (once)
   # Follow the instructions to connect to Google Drive, Dropbox, etc.
   rclone config

   # Perform backup
   rclone sync /home/$USER/important-project remote:backup-gridunesp/

   # Schedule with cron
   0 3 * * * rclone sync /home/$USER/important-results remote:backup/

Option 4: Periodic Tarball
--------------------------

.. code-block:: bash
   :caption: backup_script.sh

   #!/bin/bash
   # Script to create a compressed backup

   DATE=$(date +%Y-%m-%d)
   BACKUP_DIR="/home/$USER/backups"
   mkdir -p $BACKUP_DIR

   # List of important directories
   DIRS=(
       "/home/$USER/project1"
       "/home/$USER/project2"
       "/home/$USER/scripts"
   )

   # Create tar.gz file
   tar -czf $BACKUP_DIR/backup-$DATE.tar.gz "${DIRS[@]}"

   # Copy to a safe location (example: institutional server)
   scp $BACKUP_DIR/backup-$DATE.tar.gz user@safe-server:/backups/

   # Keep only the last 5 backups
   ls -t $BACKUP_DIR/backup-*.tar.gz | tail -n +6 | xargs -r rm

Best Practices
==============

3-2-1 Rule
----------

- **3** copies of the data
- On **2** different media
- **1** copy off-site

Backup Checklist
----------------

- [ ] Source code in Git (GitHub/GitLab)
- [ ] Important results copied to local computer
- [ ] Scripts and configurations versioned
- [ ] Raw data (if not recreatable) has an external copy
- [ ] Backup tested periodically (try restoring!)

I/O Optimization
================

For better read/write performance:

1. **Minimize the number of files**

   .. code-block:: bash

      # Bad: 10000 small files
      for i in {1..10000}; do
          echo "$i" >> file_$i.txt
      done

      # Good: 1 file
      for i in {1..10000}; do
          echo "$i"
      done > all_data.txt

2. **Use efficient binary formats**

   - HDF5 for multidimensional scientific data
   - NetCDF for geospatial/climate data
   - Parquet for large tabular data
   - NPY/NPZ for NumPy arrays

3. **Compress inactive data**

   .. code-block:: bash

      # Compress
      tar -czf data-2023.tar.gz data-2023/
      rm -rf data-2023/

      # When needed (slower, but saves space)
      tar -xzf data-2023.tar.gz

Troubleshooting
===============

**"No space left on device"**

.. code-block:: bash

   # Check usage
   df -h /home
   du -sh /home/$USER | sort -hr | head -20

   # Clean up large/unnecessary files
   find /home/$USER -name "core.*" -delete
   find /home/$USER -name "*.tmp" -delete
   find /home/$USER -name "slurm-*.out" -mtime +30 -delete

**"Permission denied" when accessing /store/**

.. code-block:: bash

   # Check permissions
   ls -la /store/$USER

   # Fix if necessary
   ### Full access for the owner (read, write and execute)
   ### and read and execute permission for group and other users
   chmod 755 /store/$USER
   ### Allows owner to read and write,
   ### and other users to read only
   chmod 644 /store/$USER/*

**Files disappeared from /tmp/**

- This is **normal and expected**. /tmp/ is temporary!
- Solution:
  * Always copy results to /home/ or /store/
  * ``job-nanny`` copies them automatically at the end of successfully completed jobs

**Slow I/O performance**

1. Check whether you are using /tmp/ for intensive I/O
2. Reduce the number of small files
3. Use binary formats instead of text
4. Avoid listing directories with many files (``ls``)

Frequently Asked Questions
==========================

**Q: Can I increase my disk quota?**

A: There are no quotas! But use the space responsibly. It is shared among all users.

**Q: How long does data stay in /store/?**

A: Permanently, until you delete it or hardware fails. **Back it up!**

**Q: Why was my job data in /store/ deleted?**

A: ``job-nanny`` automatically deletes data after jobs that finished successfully.

**Q: Can I recover accidentally deleted data?**

A: **No.** GridUnesp has no recovery system. Use backups.

**Q: How do I share data with colleagues?**

A: Use /store/ and adjust group permissions, or use /home/ with appropriate permissions.

**Q: Can I mount /home/ on my local computer?**

A: Not directly. Use scp, rsync, or sshfs for remote access.

**Q: What happens if /store/ fills up?**

A: Jobs may fail and the system may slow down. Clean up unnecessary data.

.. seealso::

   - :ref:`best_practices` - General recommendations
   - :ref:`running_simulations` - How to use the storage systems in jobs
   - :ref:`using_job_nanny` - Automatic file management
   - :ref:`transferring_files` - How to transfer data