Parallel & MPI Jobs

# Parallel & MPI Jobs

## Types of Parallelism

### Multithreading (Shared Memory)

A single process uses multiple CPU cores via threads (OpenMP, pthreads, Java threads). All threads share the same memory space and must run on the **same node**.

```bash
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=16    # 16 threads for one process
```

### MPI (Distributed Memory)

Multiple processes communicate via message passing (MPI). Processes can span **multiple nodes**, each with its own memory.

```bash
#SBATCH --ntasks=64           # 64 MPI ranks
#SBATCH --nodes=4             # Spread across 4 nodes
```

### Hybrid (MPI + Threads)

Each MPI rank spawns multiple threads. Common in modern scientific codes (GROMACS, VASP, WRF).

```bash
#SBATCH --ntasks=8            # 8 MPI ranks
#SBATCH --cpus-per-task=8     # 8 threads per rank
#SBATCH --nodes=2             # 2 nodes
```

Total CPUs = ntasks x cpus-per-task = 64

---

## MPI Job Basics

### Launching MPI Jobs with srun

Slurm's `srun` is the recommended MPI launcher. It replaces `mpirun`/`mpiexec` in most cases:

```bash
#!/bin/bash
#SBATCH --job-name=mpi_md
#SBATCH --partition=batch
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --time=2-00:00:00
#SBATCH --mem-per-cpu=2G

module load gromacs/2024

srun gmx_mpi mdrun -s topol.tpr -o traj.trr
```

`srun` automatically knows how many tasks to launch and where to place them based on the `#SBATCH` directives.

### Using mpirun Instead

Some applications or MPI implementations work better with `mpirun`. This is supported but requires more care:

```bash
#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --nodes=2
#SBATCH --ntasks=64
#SBATCH --time=04:00:00

module load openmpi/4.1

mpirun -np $SLURM_NTASKS ./my_mpi_program
```

> **Recommendation:** Use `srun` as your default MPI launcher. It integrates better with Slurm's resource management and process tracking.

---

## Controlling Task Distribution

### --ntasks-per-node

Control how many MPI ranks land on each node:

```bash
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16    # 16 ranks per node = 64 total
```

### Fill Up vs. Spread

By default, Slurm allocates tasks in a block distribution (fills up one node before moving to the next). You can change this:

```bash
# Round-robin: distribute tasks evenly across nodes
srun --distribution=cyclic my_program

# Block: fill each node before moving to next (default)
srun --distribution=block my_program
```

### Single-Node MPI

For MPI programs that should stay on one node (e.g., small shared-memory MPI jobs):

```bash
#SBATCH --nodes=1
#SBATCH --ntasks=8              # 8 ranks on a single node
```

---

## Hybrid MPI + OpenMP

A very common pattern where each MPI rank spawns OpenMP threads:

```bash
#!/bin/bash
#SBATCH --job-name=hybrid
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=4     # 4 MPI ranks per node
#SBATCH --cpus-per-task=8       # 8 OpenMP threads per rank
#SBATCH --time=12:00:00

module load gromacs/2024

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun gmx_mpi mdrun -s topol.tpr -ntomp $OMP_NUM_THREADS
```

Total resources: 4 nodes x 4 ranks x 8 threads = 128 CPUs.

---

## Life Science MPI Examples

### GROMACS Molecular Dynamics

```bash
#!/bin/bash
#SBATCH --job-name=gromacs
#SBATCH --partition=batch
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --cpus-per-task=2
#SBATCH --time=2-00:00:00
#SBATCH --mem-per-cpu=4G

module load gromacs/2024

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

srun gmx_mpi mdrun -s production.tpr \
    -ntomp $OMP_NUM_THREADS \
    -dlb yes \
    -o traj.trr \
    -e ener.edr
```

### RELION Cryo-EM Refinement (MPI + Threads)

```bash
#!/bin/bash
#SBATCH --job-name=relion_refine
#SBATCH --partition=gpu
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=5
#SBATCH --cpus-per-task=4
#SBATCH --gres=gpu:4
#SBATCH --mem=0
#SBATCH --time=3-00:00:00

module load relion/4.0

srun relion_refine_mpi \
    --i particles.star \
    --o Refine3D/run1 \
    --ref initial_model.mrc \
    --angpix 1.06 \
    --gpu "" \
    --j $SLURM_CPUS_PER_TASK
```

### NAMD Molecular Dynamics

```bash
#!/bin/bash
#SBATCH --job-name=namd
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --time=7-00:00:00
#SBATCH --mem-per-cpu=2G

module load namd/2.14

srun namd2 +ppn $SLURM_CPUS_PER_TASK production.namd > output.log
```

---

## Troubleshooting MPI Jobs

### Job Fails Immediately

1. Check that the MPI module is loaded
2. Verify `srun` is used (not a bare `mpirun` without proper setup)
3. Check the output/error files for MPI initialization errors

### Performance Issues

1. **Check task placement:** Use `srun --label` to see which node each rank is on
2. **Network:** MPI-intensive codes benefit from high-speed interconnects (InfiniBand)
3. **Memory per rank:** If ranks are competing for memory, reduce `--ntasks-per-node`
4. **Thread binding:** For hybrid jobs, ensure `OMP_NUM_THREADS` matches `--cpus-per-task`

> **ParallelCluster / PCS Note:** On AWS, **Elastic Fabric Adapter (EFA)** provides low-latency, high-bandwidth networking equivalent to InfiniBand for tightly coupled MPI workloads. Use EFA-enabled instance types (e.g., `hpc6a`, `p5`, `c5n`) and the `--constraint=efa` flag if your cluster is configured for it. Without EFA, multi-node MPI jobs will run over standard TCP and may see significantly degraded performance.

```bash
# Debug: see where tasks are placed
$ srun --label hostname
0: node001
1: node001
2: node002
3: node002
```

---

## Quick Reference

| Pattern | sbatch Flags |
|---------|-------------|
| Threaded (OpenMP) | `--ntasks=1 --cpus-per-task=N` |
| MPI across nodes | `--ntasks=N --nodes=M` |
| MPI per node | `--nodes=M --ntasks-per-node=K` |
| Hybrid MPI+OpenMP | `--ntasks=N --cpus-per-task=T --nodes=M` |
| MPI on one node | `--nodes=1 --ntasks=N` |

Exercises¶

Submit a multi-process job with --ntasks

Write a job script that requests 4 tasks (--ntasks=4) on a single node. Use srun hostname inside the script to launch 4 processes and verify that all 4 report back. Use --label to see which rank runs on which node.

Hint / Solution

cat > multi_task.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=multi_task
#SBATCH --output=multi_task_%j.out
#SBATCH --time=00:05:00
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=4G

srun --label hostname
EOF

sbatch multi_task.sh

# Output should show 4 lines like:
# 0: node042
# 1: node042
# 2: node042
# 3: node042

Submit a multithreaded job with --cpus-per-task

Write a job script for a single-process, multithreaded application. Request 1 task with 8 CPUs per task. Set OMP_NUM_THREADS from $SLURM_CPUS_PER_TASK and verify the value inside the job.

Hint / Solution

cat > threaded_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=threaded
#SBATCH --output=threaded_%j.out
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=4G

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
echo "SLURM_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK"
echo "Available processors: $(nproc)"
EOF

sbatch threaded_job.sh

Submit a hybrid MPI+OpenMP job

Write a job script that requests 4 MPI tasks with 4 CPUs per task (16 CPUs total). Set OMP_NUM_THREADS to match --cpus-per-task. Use srun --label to confirm the task layout.

Hint / Solution

cat > hybrid_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=hybrid
#SBATCH --output=hybrid_%j.out
#SBATCH --time=00:10:00
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=2G

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

echo "Total MPI ranks: $SLURM_NTASKS"
echo "Threads per rank: $OMP_NUM_THREADS"
echo "Total CPUs: $(( SLURM_NTASKS * SLURM_CPUS_PER_TASK ))"

srun --label bash -c 'echo "Rank on $(hostname), PID=$$, OMP_NUM_THREADS=$OMP_NUM_THREADS"'
EOF

sbatch hybrid_job.sh

Check task placement across nodes

Submit a job requesting 8 tasks across 2 nodes (--nodes=2 --ntasks-per-node=4). Use srun --label hostname to verify that 4 tasks land on each node.

Hint / Solution

cat > multinode.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=multinode
#SBATCH --output=multinode_%j.out
#SBATCH --time=00:05:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --mem-per-cpu=1G

srun --label hostname
EOF

sbatch multinode.sh

# Output should show ranks 0-3 on one node, ranks 4-7 on another

Parallel & MPI Jobs

Exercises¶

References¶