Skip to content

Parallel & MPI Jobs

Exercises

  1. Submit a multi-process job with --ntasks

Write a job script that requests 4 tasks (--ntasks=4) on a single node. Use srun hostname inside the script to launch 4 processes and verify that all 4 report back. Use --label to see which rank runs on which node.

Hint / Solution 
cat > multi_task.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=multi_task
#SBATCH --output=multi_task_%j.out
#SBATCH --time=00:05:00
#SBATCH --nodes=1
#SBATCH --ntasks=4
#SBATCH --mem=4G

srun --label hostname
EOF

sbatch multi_task.sh

# Output should show 4 lines like:
# 0: node042
# 1: node042
# 2: node042
# 3: node042
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sbatch multi_task.sh
Submitted batch job 42
  1. Submit a multithreaded job with --cpus-per-task

Write a job script for a single-process, multithreaded application. Request 1 task with 8 CPUs per task. Set OMP_NUM_THREADS from $SLURM_CPUS_PER_TASK and verify the value inside the job.

Hint / Solution 
cat > threaded_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=threaded
#SBATCH --output=threaded_%j.out
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=4G

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK
echo "OMP_NUM_THREADS=$OMP_NUM_THREADS"
echo "SLURM_CPUS_PER_TASK=$SLURM_CPUS_PER_TASK"
echo "Available processors: $(nproc)"
EOF

sbatch threaded_job.sh
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sbatch threaded_job.sh
Submitted batch job 42
  1. Submit a hybrid MPI+OpenMP job

Write a job script that requests 4 MPI tasks with 4 CPUs per task (16 CPUs total). Set OMP_NUM_THREADS to match --cpus-per-task. Use srun --label to confirm the task layout.

Hint / Solution 
cat > hybrid_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=hybrid
#SBATCH --output=hybrid_%j.out
#SBATCH --time=00:10:00
#SBATCH --ntasks=4
#SBATCH --cpus-per-task=4
#SBATCH --mem-per-cpu=2G

export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK

echo "Total MPI ranks: $SLURM_NTASKS"
echo "Threads per rank: $OMP_NUM_THREADS"
echo "Total CPUs: $(( SLURM_NTASKS * SLURM_CPUS_PER_TASK ))"

srun --label bash -c 'echo "Rank on $(hostname), PID=$$, OMP_NUM_THREADS=$OMP_NUM_THREADS"'
EOF

sbatch hybrid_job.sh
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sbatch hybrid_job.sh
Submitted batch job 42
  1. Check task placement across nodes

Submit a job requesting 8 tasks across 2 nodes (--nodes=2 --ntasks-per-node=4). Use srun --label hostname to verify that 4 tasks land on each node.

Hint / Solution 
cat > multinode.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=multinode
#SBATCH --output=multinode_%j.out
#SBATCH --time=00:05:00
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=4
#SBATCH --mem-per-cpu=1G

srun --label hostname
EOF

sbatch multinode.sh

# Output should show ranks 0-3 on one node, ranks 4-7 on another
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sbatch multinode.sh
Submitted batch job 42
  1. Scaling check with a CPU loop

Run the same fixed amount of pure-Python CPU work with 1, 2 and 4 CPUs and compare wall time and CPU time in sacct. The worker count comes from SLURM_CPUS_PER_TASK, never from os.cpu_count().

Hint / Solution 
cat > busy.py << 'EOF'
import os, multiprocessing as mp
N = int(os.environ.get("SLURM_CPUS_PER_TASK", "1"))
TOTAL = 40_000_000          # fixed total work, split across N workers
def work(n):
    s = 0
    for i in range(n):
        s += i * i
    return s
if __name__ == "__main__":
    with mp.Pool(N) as p:
        p.map(work, [TOTAL // N] * N)
EOF

for c in 1 2 4; do
  sbatch --job-name=scale${c} --cpus-per-task=${c} --ntasks=1 --time=00:05:00 \
    --output=scale${c}_%j.out --wrap="python3 busy.py"
done

# After all three finish:
sacct -X --name=scale1,scale2,scale4 -o JobID,JobName,AllocCPUS,TotalCPU,Elapsed
`Elapsed` should drop roughly in proportion to `AllocCPUS` (up to the physical cores available), while `TotalCPU` stays roughly constant because the total work is fixed. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sbatch --job-name=val_scale1 --cpus-per-task=1 --ntasks=1 --time=00:05:00          --output=scale1_%j.out --wrap='/shared/python/3.12/bin/python3 busy.py'
Submitted batch job 42

$ sbatch --job-name=val_scale2 --cpus-per-task=2 --ntasks=1 --time=00:05:00          --output=scale2_%j.out --wrap='/shared/python/3.12/bin/python3 busy.py'
Submitted batch job 43

$ sbatch --job-name=val_scale4 --cpus-per-task=4 --ntasks=1 --time=00:05:00          --output=scale4_%j.out --wrap='/shared/python/3.12/bin/python3 busy.py'
Submitted batch job 44

$ sacct -X --name=val_scale1,val_scale2,val_scale4 -o JobID,JobName,AllocCPUS,TotalCPU,Elapsed
JobID           JobName  AllocCPUS   TotalCPU    Elapsed 
------------ ---------- ---------- ---------- ---------- 
86           val_scale1          1   00:00:00   00:00:06 
87           val_scale2          2   00:00:00   00:00:05 
88           val_scale4          4   00:00:00   00:00:02 
42          val_scale1          1   00:00:00   00:00:04 
43          val_scale2          2   00:00:00   00:00:03 
44          val_scale4          4   00:00:00   00:00:02

References