Resource Requests

<!-- audience: L2, L3 --> <!-- deployment: all --> <!-- sge-source: SGE "Requesting Resources" + "Exclusive Access" --> # Resource Requests ## The Basics Every job competes for shared resources. The more precisely you describe what your job needs, the faster the scheduler can place it and the less waste there is for everyone. Slurm's resource model is built around: **CPUs, memory, time, nodes, and generic resources (GRES)**. --- ## CPUs: --ntasks vs --cpus-per-task This is the most common source of confusion in Slurm. There are two distinct ways to request CPUs: ### --ntasks (-n): Number of Processes Use when your program launches **multiple independent processes** (e.g., MPI ranks): ```bash #SBATCH --ntasks=16 # 16 MPI ranks ``` ### --cpus-per-task (-c): Threads per Process Use when your program uses **multithreading** within a single process (e.g., OpenMP, Java threads, Python multiprocessing): ```bash #SBATCH --ntasks=1 # One process #SBATCH --cpus-per-task=16 # 16 threads for that process ``` ### Decision Guide | Your program... | Use | |-----------------|-----| | Is a serial program (single-threaded) | `--ntasks=1` (default) | | Uses threads (OpenMP, pthreads) | `--ntasks=1 --cpus-per-task=N` | | Uses MPI | `--ntasks=N` | | Uses MPI + threads (hybrid) | `--ntasks=N --cpus-per-task=M` | > **Common mistake:** Using `--ntasks=16` for a threaded program. This tells Slurm you want 16 separate processes, not 16 threads for one process. The result: 16 copies of your program, each using 1 CPU. --- ## Memory ### Per Node ```bash #SBATCH --mem=64G # 64 GB total for the job on each node ``` ### Per CPU ```bash #SBATCH --mem-per-cpu=4G # 4 GB per allocated CPU ``` ### Which to Use? - `--mem` is simpler for single-node jobs - `--mem-per-cpu` scales naturally with CPU count, good for MPI jobs across multiple nodes - Don't specify both -- they're mutually exclusive ### Checking Memory After a Job ```bash $ sacct -j 12345 --format=JobID,ReqMem,MaxRSS,State JobID ReqMem MaxRSS State -------- ------ ------ --------- 12345 64Gn COMPLETED 12345.batch 64Gn 28456M COMPLETED ``` If `MaxRSS` is much less than `ReqMem`, you're over-requesting. If your job was killed with `OOM`, you're under-requesting. --- ## Time Limits (--time) Every job should specify a wall-clock time limit. This is **required on most clusters** and is a key difference from SGE. ```bash #SBATCH --time=04:00:00 # 4 hours #SBATCH --time=2-12:00:00 # 2 days, 12 hours #SBATCH --time=30:00 # 30 minutes ``` ### Accepted Formats | Format | Example | Duration | |--------|---------|----------| | `MM:SS` | `30:00` | 30 minutes | | `HH:MM:SS` | `04:00:00` | 4 hours | | `D-HH:MM:SS` | `2-12:00:00` | 2 days, 12 hours | | `D-HH` | `7-00` | 7 days | ### Why Time Limits Matter - Jobs exceeding their time limit are **killed** (state `TIMEOUT`) - Shorter time limits improve scheduling -- the backfill scheduler can fit short jobs into gaps - Request enough time with some margin, but don't pad excessively > **Tip:** Check your past jobs with `sacct` to learn typical runtimes, then add 20-50% buffer. --- ## Partitions (-p) Partitions group nodes by capability or purpose. Request a partition to direct your job to appropriate hardware: ```bash #SBATCH --partition=gpu # GPU-equipped nodes #SBATCH --partition=highmem # Large-memory nodes #SBATCH --partition=batch # General compute (often the default) ``` If you don't specify a partition, Slurm uses the cluster's default partition (marked with `*` in `sinfo` output). You can specify multiple partitions -- Slurm will use whichever can start your job first: ```bash #SBATCH --partition=batch,long ``` --- ## Exclusive Node Access When your job needs an entire node with no other jobs sharing it: ```bash #SBATCH --exclusive ``` Use cases: - Benchmarking (need consistent performance without interference) - Programs that use all available memory - Security-sensitive workloads With `--exclusive`, you get all CPUs and GRES on the node, but only the memory you requested. To get all memory too: ```bash #SBATCH --exclusive #SBATCH --mem=0 # Request ALL memory on the node ``` --- ## Generic Resources (GRES) GRES are administrator-defined resources like GPUs, FPGAs, or software licenses. ### GPUs ```bash #SBATCH --gres=gpu:1 # 1 GPU (any type) #SBATCH --gres=gpu:a100:2 # 2 NVIDIA A100 GPUs #SBATCH --gres=gpu:v100:4 # 4 NVIDIA V100 GPUs ``` Slurm also provides GPU-specific shorthand options: ```bash #SBATCH --gpus=2 # 2 GPUs per job #SBATCH --gpus-per-node=1 # 1 GPU per node #SBATCH --gpus-per-task=1 # 1 GPU per task ``` Associated GPU options: ```bash #SBATCH --cpus-per-gpu=4 # 4 CPUs per GPU allocated #SBATCH --mem-per-gpu=32G # 32 GB memory per GPU ``` ### Software Licenses If your cluster tracks licenses as GRES: ```bash #SBATCH --gres=license:matlab:1 ``` Or via the `--licenses` flag: ```bash #SBATCH --licenses=matlab:1 ``` --- ## Constraints and Features Administrators can tag nodes with **features** (e.g., CPU architecture, interconnect type, local SSD). You can require specific features: ```bash #SBATCH --constraint=avx512 # Require AVX-512 capable nodes #SBATCH --constraint="intel&ssd" # Require both Intel CPU AND local SSD #SBATCH --constraint="a100|v100" # Accept either A100 OR V100 nodes ``` --- ## Putting It Together: Common Patterns ### Serial Bioinformatics Job ```bash #SBATCH --ntasks=1 #SBATCH --cpus-per-task=1 #SBATCH --mem=8G #SBATCH --time=02:00:00 ``` ### Multithreaded Analysis (e.g., BLAST, samtools) ```bash #SBATCH --ntasks=1 #SBATCH --cpus-per-task=16 #SBATCH --mem=32G #SBATCH --time=04:00:00 ``` --- ## Common Patterns: GPU Deep Learning ### GPU Deep Learning Job ```bash #SBATCH --partition=gpu #SBATCH --ntasks=1 #SBATCH --cpus-per-task=8 #SBATCH --mem=64G #SBATCH --gres=gpu:1 #SBATCH --time=24:00:00 ``` --- ## Common Patterns: MPI and High-Memory ### MPI Molecular Dynamics (multi-node) ```bash #SBATCH --partition=batch #SBATCH --nodes=4 #SBATCH --ntasks-per-node=32 #SBATCH --mem-per-cpu=2G #SBATCH --time=2-00:00:00 ``` ### High-Memory Genome Assembly ```bash #SBATCH --partition=highmem #SBATCH --ntasks=1 #SBATCH --cpus-per-task=32 #SBATCH --mem=500G #SBATCH --time=3-00:00:00 ``` > **ParallelCluster Note:** Resource availability depends on the instance types configured in your cluster. Check `sinfo -N -l` to see what's available, and note that dynamic nodes may take a few minutes to launch. --- ## Right-Sizing Your Jobs Over-requesting wastes resources and can increase your wait time. Under-requesting causes failures. Review completed jobs to calibrate: ```bash # Check actual resource usage of a completed job $ sacct -j 12345 --format=JobID,Elapsed,MaxRSS,MaxVMSize,TotalCPU,AllocCPUS # Compare requested vs actual $ seff 12345 # (if installed on your cluster) Job ID: 12345 Cluster: mycluster State: COMPLETED Cores: 16 CPU Utilized: 12:34:56 CPU Efficiency: 78.5% Memory Utilized: 24.5 GB Memory Efficiency: 76.6% of 32.00 GB ```

Exercises

1. Request specific memory and CPUs for a threaded job

Write a job script that simulates a multithreaded bioinformatics tool: request 1 task, 4 CPUs per task, and 8G of memory. Inside the script, print the values of $SLURM_CPUS_PER_TASK and $SLURM_MEM_PER_NODE to confirm the allocation.

Hint / Solution

cat > resource_test.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=resource_test
#SBATCH --output=resource_test_%j.out
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=4
#SBATCH --mem=8G

echo "CPUs per task: $SLURM_CPUS_PER_TASK"
echo "Memory per node (MB): $SLURM_MEM_PER_NODE"
echo "Number of tasks: $SLURM_NTASKS"
EOF

sbatch resource_test.sh

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch resource_test.sh
Submitted batch job 42

1. Request exclusive node access

Submit a job with --exclusive and --mem=0 to claim an entire node. Inside the job, run nproc to see how many CPUs you received and free -g to check available memory.

Hint / Solution

sbatch --exclusive --mem=0 --time=00:10:00 --job-name=exclusive_test \
    --output=exclusive_%j.out --wrap="echo CPUs: \$(nproc); free -g"

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --exclusive --mem=0 --time=00:10:00 --job-name=val_rr2        --output=exclusive_%j.out --wrap='echo CPUs: $(nproc); free -g'
Submitted batch job 42

1. Use --constraint to request a node feature

Check what features are defined on your cluster's nodes using sinfo -o "%n %f". Then submit a job that requires a specific feature using --constraint. If no features are defined, try the exercise conceptually and write the #SBATCH line you would use.

Hint / Solution

# Discover available features
sinfo -o "%n %f"

# Submit with a constraint (replace "avx512" with an actual feature on your cluster)
sbatch --constraint=avx512 --time=00:05:00 --wrap="echo 'Running on AVX-512 node'"

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sinfo -o '%n %f'
HOSTNAMES AVAIL_FEATURES
batch-dy-compute-3 dynamic,t3.2xlarge,compute
batch-dy-compute-4 dynamic,t3.2xlarge,compute
batch-dy-compute-1 dynamic,t3.2xlarge,compute
batch-dy-compute-2 dynamic,t3.2xlarge,compute
debug-dy-small-1 dynamic,t3.medium,small
debug-dy-small-2 dynamic,t3.medium,small
gpu-dy-t4-1 dynamic,g4dn.xlarge,t4,gpu
highmem-dy-large-1 dynamic,t3.2xlarge,large

1. Check actual resource usage after a job completes

Submit a job that allocates 16G of memory but only uses a small amount (e.g., run sleep 30). After it completes, use sacct to compare requested memory (ReqMem) vs. actual peak usage (MaxRSS). Was the job over-provisioned?

Hint / Solution

sbatch --time=00:05:00 --mem=16G --wrap="sleep 30" --job-name=mem_check

# After completion:
sacct -j <jobid> --format=JobID,JobName,ReqMem,MaxRSS,Elapsed,State

# MaxRSS on the .batch step will be much less than 16G,
# indicating over-provisioning. For real workloads, use this
# technique to right-size your future requests.

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --time=00:05:00 --mem=16G --job-name=val_rr4 --output=/dev/null --wrap='sleep 10'
Submitted batch job 42

$ sacct -j 42 --format=JobID,JobName,ReqMem,MaxRSS,Elapsed,State
JobID           JobName     ReqMem     MaxRSS    Elapsed      State 
------------ ---------- ---------- ---------- ---------- ---------- 
42             val_rr4        16G              00:00:11  COMPLETED 
42.batch         batch                  904K   00:00:11  COMPLETED 

1. Compare --mem vs --mem-per-cpu

Submit two jobs that each request the equivalent of 16G total memory but using different flags: one with --mem=16G and one with --mem-per-cpu=4G --cpus-per-task=4. Use scontrol show job on each to compare how Slurm records the memory request.

Hint / Solution

JOB1=$(sbatch --parsable --time=00:05:00 --mem=16G --cpus-per-task=4 --wrap="sleep 60")
JOB2=$(sbatch --parsable --time=00:05:00 --mem-per-cpu=4G --cpus-per-task=4 --wrap="sleep 60")

scontrol show job $JOB1 | grep MinMemory
scontrol show job $JOB2 | grep MinMemory
# The first shows MinMemoryNode=16G, the second shows MinMemoryCPU=4G

scancel $JOB1 $JOB2

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --parsable --hold --time=00:05:00 --mem=16G --cpus-per-task=4 --job-name=val_rr5a --wrap='sleep 60'
695

$ sbatch --parsable --hold --time=00:05:00 --mem-per-cpu=4G --cpus-per-task=4 --job-name=val_rr5b --wrap='sleep 60'
696

$ scontrol show job 695 | grep -E '(MinMemory|TRES=)'
   ReqTRES=cpu=4,mem=16G,node=1,billing=4
   AllocTRES=(null)
   MinCPUsNode=4 MinMemoryNode=16G MinTmpDiskNode=0

$ scontrol show job 696 | grep -E '(MinMemory|TRES=)'
   ReqTRES=cpu=4,mem=16G,node=1,billing=4
   AllocTRES=(null)
   MinCPUsNode=4 MinMemoryCPU=4G MinTmpDiskNode=0

$ scancel 695 696

References

• SchedMD: sbatch man page -- Resource Options
• SchedMD: Generic Resource (GRES) Scheduling
• SchedMD: CPU Management Guide
• SchedMD: Consumable Resources