Shared Memory

Shared memory is a parallelism model in which multiple threads (lines of execution) share the same memory space within a single node (server).

Basic Concepts

  • Threads: Multiple lines of execution within the same process

  • Shared memory: All threads see the same memory

  • OpenMP: The most common standard for shared-memory programming

  • SMP (Symmetric Multi-Processing): Architecture with multiple processors sharing memory

Characteristics:

  • All cores used are on the same node

  • Access to the same memory area

  • Communication between threads is very fast

  • Limited to the resources of a single node

OpenMP

OpenMP is an API for parallel programming in C/C++ and Fortran, based on compiler directives.

Example in C

omp_hello.c
/******************************************************************************
 * OpenMP Example - Hello World
 * The master thread creates a parallel region.
 * All threads obtain their unique number and print.
 ******************************************************************************/
#include <omp.h>
#include <stdio.h>
#include <stdlib.h>

int main (int argc, char *argv[])
{
  int nthreads, tid;

  /* Fork: create a team of threads */
  #pragma omp parallel private(nthreads, tid)
  {
    /* Get the thread number */
    tid = omp_get_thread_num();
    printf("Hello World from thread = %d\n", tid);

    /* Only the master thread does this */
    if (tid == 0)
    {
      nthreads = omp_get_num_threads();
      printf("Number of threads = %d\n", nthreads);
    }

  }  /* Join: all threads meet here */

  return 0;
}

Compiling

On GridUnesp, we recommend the Intel compiler, but GCC is also available:

# With Intel
module load intel/compilers
icc -openmp omp_hello.c -o omp_hello

# With GCC
module load gcc/10.2.0
gcc -fopenmp omp_hello.c -o omp_hello

Submission Script

In SLURM, the number of threads is controlled by the -c (--cpus-per-task) directive.

job_openmp.sh
#!/bin/bash
#SBATCH -J openmp_test
#SBATCH -N 1                # 1 node (required for shared memory)
#SBATCH -c 8                # 8 threads
#SBATCH -t 01:00:00
#SBATCH --mem=16G

export INPUT="input.dat"
export OUTPUT="output_openmp.dat"

# Set the number of threads for OpenMP
export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

# Load the compiler module
module load intel/compilers

job-nanny srun ./omp_hello

Important Variables

  • OMP_NUM_THREADS: Number of threads to be used

  • OMP_STACKSIZE: Stack size per thread

  • OMP_SCHEDULE: Type of loop scheduling

export OMP_NUM_THREADS=28
export OMP_STACKSIZE=256M
export OMP_SCHEDULE="dynamic"

Limitations

  • Limited to 1 node (cannot use multiple servers)

  • Maximum threads per node: 56 (28 physical cores + hyper-threading)

  • Maximum available threads per node: 52 (26 physical cores + hyper-threading)

  • Total memory limited to the node’s RAM: 128 GB

Cluster Resource Allocation (Core Specialization)

Node Type

Total Capacity

Reserved for OS

Available for Jobs

Regular Nodes (node[001-056])

28 Cores / 56 CPUs

2 Cores / 4 CPUs

26 Cores / 52 CPUs

GPU Node (gpunode001)

48 Cores / 96 CPUs

4 Cores / 8 CPUs

44 Cores / 88 CPUs

When to Use Shared Memory

✅ Ideal for:

  • Problems that fit in the memory of a single node

  • Heavy loops that can be parallelized

  • Applications with a lot of communication between threads

  • When network latency would be a problem

❌ Not recommended for:

  • Problems that do not fit in 128 GB of RAM

  • Applications that need more than 52 threads

  • When the problem is naturally distributed

Practical Example: Matrix Product

matmul_omp.c
#include <stdio.h>
#include <stdlib.h>
#include <omp.h>

#define N 1000

int main() {
    double A[N][N], B[N][N], C[N][N];
    double start, end;

    // Initialization
    #pragma omp parallel for
    for (int i = 0; i < N; i++) {
        for (int j = 0; j < N; j++) {
            A[i][j] = i + j;
            B[i][j] = i - j;
            C[i][j] = 0.0;
        }
    }

    start = omp_get_wtime();

    // Parallel multiplication
    #pragma omp parallel for
    for (int i = 0; i < N; i++) {
        for (int k = 0; k < N; k++) {
            for (int j = 0; j < N; j++) {
                C[i][j] += A[i][k] * B[k][j];
            }
        }
    }

    end = omp_get_wtime();
    printf("Time with %d threads: %f seconds\n",
           omp_get_max_threads(), end - start);

    return 0;
}

Script for a scalability test:

test_omp_scaling.sh
#!/bin/bash
# Test with 1, 2, 4, 8, 16, 28, 52 threads
for threads in 1 2 4 8 16 28 52; do
    cat > job_${threads}.sh << EOF
#!/bin/bash
#SBATCH -J omp_${threads}
#SBATCH -N 1
#SBATCH -c ${threads}
#SBATCH -t 00:30:00
#SBATCH --mem=16G
#SBATCH --output=omp_${threads}_%j.out

export OMP_NUM_THREADS=${threads}
module load intel/compilers
./test_omp
EOF

    sbatch job_${threads}.sh
done

Optimization Tips

  1. Load balancing: Use dynamic scheduling for irregular loops

    #pragma omp parallel for schedule(dynamic, 100)

  2. Avoid false sharing: Place shared variables on different cache lines

  3. Use private variables: Declare temporary variables as private

    #pragma omp parallel for private(temp)

  4. Nested parallelism: Be careful when nesting parallel regions

    export OMP_NESTED=FALSE

See also