Best Practices

<!-- audience: L2, L3 --> <!-- deployment: all --> <!-- sge-source: SGE "Effective Usage" tips + scattered SGE best practices --> # Best Practices ## Resource Estimation ### Right-Size Your Jobs Over-requesting resources wastes cluster capacity and can increase your wait time (larger requests are harder to schedule). Under-requesting causes job failures. **The workflow:** 1. Run a small test with generous resources 2. Check actual usage with `sacct` or `seff` 3. Adjust your requests with ~20-30% headroom 4. Repeat as you scale up ```bash # Check actual usage of a completed job $ sacct -j 12345 --format=JobID,Elapsed,MaxRSS,TotalCPU,AllocCPUS,State # seff gives a nice summary (if installed) $ seff 12345 Job ID: 12345 State: COMPLETED (exit code 0) Cores: 16 CPU Utilized: 10:23:45 CPU Efficiency: 64.8% of 16:00:00 core-walltime Memory Utilized: 24.5 GB Memory Efficiency: 76.6% of 32.00 GB ``` --- ## Right-size with evidence 1. Run a test job with a generous request. 2. Read what it actually used: ```bash sacct -j <jobid> --format=JobID,ReqMem,MaxRSS,AllocCPUS,TotalCPU,Elapsed ``` 3. Request `MaxRSS` plus 20-30% headroom, and a `--time` a bit above `Elapsed`. 4. Optimise for **time-to-solution = queue wait + run time**, not run time alone: a smaller request often starts sooner. > `--cpus-per-task>1` on code that is not threaded wastes cores and lowers your fair-share priority for the next job. **26.05+:** `sbatch --mem-update=<margin>@<delay>` lowers a running job's memory limit to current usage plus `margin` percent after `delay` (e.g. `--mem-update=25@10:00`). It only reduces, never raises, and needs `enable_stepmgr` and `jobacct_gather/cgroup` on the cluster. --- ## Reading `seff` and `reportseff` `seff <jobid>` ships in Slurm contribs and may be absent on your cluster. Read it field by field: | Field | Meaning | Act on it | |-------|---------|-----------| | `Cores` | CPUs allocated | Compare with CPU Efficiency | | `CPU Utilized` | `TotalCPU`: CPU time actually used | | | `CPU Efficiency` | CPU Utilized / (cores × wall time) | Low on a multi-core job: request fewer cores | | `Memory Utilized` | Peak RSS | Request this + 20-30% | | `Memory Efficiency` | Utilized / requested | Under ~50%: lower `--mem` | For arrays or filtered views use [reportseff](https://github.com/troycomi/reportseff): ```bash pip install --user reportseff reportseff -u $USER --since d=1 # one row per job, colour-coded reportseff 12345 # array tasks shown individually ``` --- ## Time Limits and Memory ### Time Limits - **Too short:** Job gets killed (TIMEOUT), wasting all compute time spent - **Too long:** Reduces your priority for backfill scheduling - **Sweet spot:** Actual expected time + 30-50% buffer ```bash # Check runtimes of your recent similar jobs $ sacct --name=blast_run --format=JobID,Elapsed,State --starttime=now-30days ``` ### Memory - Check `MaxRSS` from `sacct` -- this is peak actual memory usage - Request 20-30% more than `MaxRSS` for safety - If you don't know, start with the node's per-CPU default and adjust --- ## Job Submission Practices ### Use Job Arrays, Not Loops ```bash # BAD for i in $(seq 1 500); do sbatch process.sh $i done # GOOD sbatch --array=1-500%50 process.sh ``` ### Use Dependencies for Pipelines ```bash # BAD: manual waiting and checking sbatch step1.sh # ... manually check, then ... sbatch step2.sh # GOOD: automated pipeline JOB1=$(sbatch --parsable step1.sh) sbatch --dependency=afterok:$JOB1 step2.sh ``` ### Name Your Jobs ```bash #SBATCH --job-name=blast_sampleA ``` Good job names make `squeue` output readable and `sacct` queries useful. --- ## Job Submission Practices (cont.) ### Organize Output Files ```bash #SBATCH --output=logs/%x_%j.out #SBATCH --error=logs/%x_%j.err ``` Create the `logs/` directory before submitting. Use `%x` (job name) and `%j` (job ID) for unique filenames. ### Set Mail Notifications Wisely ```bash #SBATCH --mail-type=END,FAIL # Notified when job ends or fails #SBATCH --mail-user=you@example.com ``` Don't use `--mail-type=ALL` for array jobs -- you'll get thousands of emails. --- ## Job Script Best Practices ### Make Scripts Self-Contained ```bash #!/bin/bash #SBATCH --job-name=my_analysis #SBATCH --time=04:00:00 #SBATCH --cpus-per-task=8 #SBATCH --mem=32G # Clean module environment module purge module load blast/2.15 # Move to working directory (explicit) cd $SLURM_SUBMIT_DIR # Use Slurm variables for thread count blastn -query input.fasta -db /data/nt \ -num_threads $SLURM_CPUS_PER_TASK \ -out results.txt ``` --- ## Error Handling and Variables ### Use set -e for Fail-Fast ```bash #!/bin/bash #SBATCH ... set -euo pipefail # Exit on error, undefined vars, pipe failures module purge module load samtools/1.17 samtools sort input.bam -o sorted.bam samtools index sorted.bam ``` Without `set -e`, the script continues after failures, producing incomplete or wrong results. ### Don't Hardcode Thread Counts ```bash # BAD blastn -num_threads 16 ... # GOOD blastn -num_threads $SLURM_CPUS_PER_TASK ... ``` This way, changing `--cpus-per-task` automatically adjusts the program's thread count. --- ## Filesystem Best Practices ### Know Your Filesystems | Filesystem | Best For | Watch Out | |-----------|----------|-----------| | `/home` | Scripts, configs, small files | Often quota-limited (10-50 GB) | | `/scratch` | Active job data, large files | Semantics vary by deployment — see below | | `/data` or `/shared` | Reference databases, shared data | Usually read-heavy | | Per-node local (`/tmp`, `$SLURM_TMPDIR`) | Temporary I/O-intensive single-node work | Not visible to other nodes; lost when the job ends | --- ### Node-local scratch vs shared scratch `/scratch` means two very different things depending on how the cluster was built. Understanding which flavor your cluster provides — and choosing the right one for your job — is often the difference between an analysis that runs in 30 minutes and one that runs in 8 hours. **Shared scratch** is a single filesystem that every node mounts. Writing a file on the head node makes it immediately visible on every compute node, and vice-versa. Most on-prem sites implement this with a parallel filesystem such as Lustre, GPFS, or BeeGFS. It's what lets a multi-node MPI job checkpoint in parallel, or a pipeline's second stage read what the first stage produced on a different node. **Node-local scratch** is storage that lives on a single node — usually a fast NVMe SSD — and is wiped when the job ends or the node stops. No other node can see it. It's roughly 10–100× faster than any network filesystem for small-file or random-access workloads, so it's the right place to stage data for compute-bound single-node jobs. Cloud HPC platforms use node-local scratch by default. **AWS ParallelCluster and AWS PCS both mount the instance's local NVMe at `/scratch` automatically on instance types that ship with instance storage** (g4dn, c5d, m5d, i3, etc.). That `/scratch` exists only while that specific EC2 instance is running — when the compute job finishes and the node scales back down, everything in it is gone. For cross-node data sharing in an AWS cluster, use a shared filesystem such as FSx for Lustre or EFS (see `deployment/aws-parallelcluster.md` and `deployment/aws-pcs.md` for the full ephemeral-storage deep dive). **When node-local scratch is the right choice:** - Single-node jobs with heavy random I/O or small-file workloads (BLAST against a local database, ML training against a dataset that fits, genome assembly scratch space). - Intermediate files you don't need after the job ends (sort intermediates, Spark shuffle output, trimmed read pairs). - When the fastest possible I/O matters and the dataset fits on the local disk. Stage in from shared storage, compute, stage out results, let the rest evaporate. - CryoSPARC SSD cache — `cache_path: /scratch/cryosparc_cache` is the canonical setup for this exact reason. ```bash #!/bin/bash #SBATCH --job-name=blast #SBATCH --time=02:00:00 #SBATCH --cpus-per-task=8 #SBATCH --mem=32G # Stage input to node-local scratch for fast random access cp /shared/databases/nr.fasta $SLURM_TMPDIR/ cp /shared/input/query.fa $SLURM_TMPDIR/ cd $SLURM_TMPDIR blastn -db nr.fasta -query query.fa -out results.tsv -num_threads 8 # Stage result back to shared storage; $SLURM_TMPDIR is wiped automatically cp results.tsv /shared/output/ ``` **When you need shared scratch instead:** - Multi-node MPI jobs that coordinate checkpoints or intermediate I/O across ranks on different nodes. - Nextflow / Snakemake / Cromwell pipelines where a later task reads what an earlier task produced on a different compute node. - Output that must survive the job (e.g., team collaboration on a long-running simulation's intermediate state). - Any time you'd otherwise write code like "copy this file from node A to node B" — a shared filesystem does that work for you. ```bash #!/bin/bash #SBATCH --job-name=gromacs_md #SBATCH --nodes=4 #SBATCH --ntasks-per-node=32 # /shared (or /fsx if you have FSx for Lustre) is visible on every node mkdir -p /shared/$USER/md_run_001 cd /shared/$USER/md_run_001 srun gmx_mpi mdrun -deffnm production # writes checkpoints every step; # all ranks on all nodes must see them ``` **Portability rule of thumb:** if your script references `/scratch`, ask yourself whether a second job running on a different node would need to see those files. If yes, use `/shared` (or the equivalent shared filesystem on your cluster). If no, `$SLURM_TMPDIR` is the portable way to ask Slurm for per-node scratch — it works on pcluster, PCS, and on-prem clusters alike. --- ## Filesystem I/O Patterns ### Direct I/O to Appropriate Storage ```bash #!/bin/bash #SBATCH ... # Copy input to local scratch for fast I/O cp /data/large_input.bam /tmp/input.bam # Run analysis on local scratch samtools sort /tmp/input.bam -o /tmp/sorted.bam # Copy results back cp /tmp/sorted.bam $SLURM_SUBMIT_DIR/sorted.bam # Clean up rm /tmp/input.bam /tmp/sorted.bam ``` ### Don't Write Many Small Files to Shared Filesystems Shared filesystems (NFS, Lustre) struggle with millions of small files. If your workflow creates many temp files, use local storage. > **ParallelCluster / PCS Note (Cloud Cost):** On cloud clusters, **accurate `--time` limits directly affect cost**. Overly generous wall times keep dynamic nodes running (and billing) longer than necessary. Conversely, be aware that scale-to-zero clusters have a **startup delay** (1-3 minutes) when new nodes must launch -- factor this into your expectations but not your `--time` request (Slurm starts the clock after the node is ready). --- ## Being a Good Cluster Citizen ### Throttle Array Jobs ```bash #SBATCH --array=1-10000%50 # Max 50 concurrent ``` ### Don't Request Exclusive Unless Needed `--exclusive` prevents other jobs from sharing your node, even if you're only using 4 of 64 CPUs. ### Cancel Jobs You Don't Need ```bash # Don't leave broken or stalled jobs running $ scancel 12345 ``` ### Test Before Scaling 1. Run one job interactively to verify it works 2. Submit a small batch (5-10 jobs) 3. Check results and resource usage 4. Scale up with confidence --- ## Scheduler etiquette ### Don't Hammer the Controller Every `squeue`/`sacct`/`sbatch` call is an RPC to `slurmctld` or `slurmdbd`. Loops of them slow the scheduler for everyone. - Poll no more than once or twice a minute; prefer blocking over polling: ```bash sbatch --wait job.sh # returns when the job ends sbatch --dependency=afterok:12345 next.sh # let Slurm chain jobs ``` ### Login Nodes Are Shared - No heavy compute, large transfers or long builds on the login node; use `srun`/`salloc`. - Run from **scratch or shared project storage, not `$HOME`** (small quota, shared by everyone). - Large executable used by many nodes? Broadcast it once instead of having every node read it from shared storage: ```bash sbcast my.prog /tmp/my.prog && srun /tmp/my.prog # or: srun --bcast=/tmp/my.prog my.prog ``` --- ## Debugging Checklist When a job fails: 1. **Check output/error files:** `cat slurm-<jobid>.out` 2. **Check exit code:** `sacct -j <jobid> --format=JobID,State,ExitCode` 3. **Check resource usage:** Did it run out of memory (OOM) or time (TIMEOUT)? 4. **Test interactively:** `srun --pty bash`, then run the same commands 5. **Check the environment:** Are all modules loaded? Are paths correct? 6. **Check permissions:** Can the compute node access your files? --- ## Quick Reference: Common Mistakes | Mistake | Consequence | Fix | |---------|-------------|-----| | No `--time` | Uses partition default (may be short) | Always set `--time` | | `--ntasks=16` for threaded app | 16 copies of program, 1 CPU each | `--ntasks=1 --cpus-per-task=16` | | No `module purge` in script | Inconsistent environment | Add `module purge` at top | | Hardcoded thread count | Doesn't match allocation | Use `$SLURM_CPUS_PER_TASK` | | Submission loop instead of array | Overloads scheduler | Use `--array` | | Output dir doesn't exist | Job fails immediately | `mkdir -p logs/` before submit | | No `--mem` specified | Gets partition default (may be low) | Always set `--mem` |

Exercises

1. Estimate resources for a job

Submit a short test job (a small dataset or a quick analysis), then use sacct to see how many resources it actually consumed. Compare the actual usage to what you requested. Were you close, or did you over- or under-request?

Hint / Solution

# Submit a test job (adjust the command to your workload)
sbatch --job-name=resource_test --time=00:30:00 --ntasks=1 \
  --cpus-per-task=4 --mem=8G --wrap="your_analysis_command input.dat"

# After the job completes, check actual resource usage
sacct -j <jobid> --format=JobID,JobName,Elapsed,MaxRSS,MaxVMSize,CPUTime,ReqMem,ReqCPUS,State

# Interpretation:
#   Elapsed    = wall-clock time (compare to --time)
#   MaxRSS     = peak memory used (compare to --mem)
#   CPUTime    = total CPU seconds consumed
#   ReqMem     = what you requested
#   ReqCPUS    = CPUs you requested

If `MaxRSS` is 2G but you requested 8G, you are wasting 6G that other users could use. If `Elapsed` is 5 minutes but you requested 30, future jobs can use a tighter `--time` to improve scheduling priority. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sacct -j 42 --format=JobID,JobName,Elapsed,MaxRSS,MaxVMSize,CPUTime,ReqMem,ReqCPUS,State
JobID           JobName    Elapsed     MaxRSS  MaxVMSize    CPUTime     ReqMem  ReqCPUS      State 
------------ ---------- ---------- ---------- ---------- ---------- ---------- -------- ---------- 
42             val_bp1   00:00:30                         00:02:00         8G        4  COMPLETED 
42.batch         batch   00:00:30       952K          0   00:02:00                   4  COMPLETED 

1. Compare resource efficiency

Submit the same short job twice with different resource requests -- once with generous resources and once with a tighter fit based on your measurements from Exercise 1. Compare the wait time and efficiency of each.

Hint / Solution

# Run 1: Generous resources
sbatch --job-name=generous --time=02:00:00 --ntasks=1 \
  --cpus-per-task=8 --mem=32G --wrap="your_analysis_command input.dat"

# Run 2: Right-sized resources (based on actual usage from Exercise 1)
sbatch --job-name=rightsized --time=00:30:00 --ntasks=1 \
  --cpus-per-task=4 --mem=8G --wrap="your_analysis_command input.dat"

# Compare both jobs after completion
sacct -j <jobid1>,<jobid2> \
  --format=JobID,JobName,Elapsed,MaxRSS,ReqMem,ReqCPUS,CPUTime,State

# Check how long each waited in the queue
sacct -j <jobid1>,<jobid2> \
  --format=JobID,JobName,Submit,Start,Elapsed,State

The right-sized job often starts sooner because the scheduler can fit it into smaller gaps in the cluster. Over hundreds of jobs (e.g., in an array), this difference adds up significantly. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sacct -j 42,43 --format=JobID,JobName,Elapsed,MaxRSS,ReqMem,ReqCPUS,CPUTime,State -X
JobID           JobName    Elapsed     MaxRSS     ReqMem  ReqCPUS    CPUTime      State 
------------ ---------- ---------- ---------- ---------- -------- ---------- ---------- 
42_2           val_ja3   00:00:10                  256M        1   00:00:10 CANCELLED+ 
42_3           val_ja3   00:00:10                  256M        1   00:00:10 CANCELLED+ 
42            val_bp2a   00:00:16                   24G        8   00:02:08  COMPLETED 
43            val_bp2b   00:00:15                    8G        4   00:01:00  COMPLETED 

1. Compute job efficiency by hand

Pick one completed job and compute its CPU efficiency from sacct fields: TotalCPU / (AllocCPUS × Elapsed). Then compare with what seff reports.

Hint / Solution

# Pick a finished job and pull the raw fields (-X = allocation line only)
sacct -j <jobid> -X --format=JobID,AllocCPUS,Elapsed,TotalCPU,State

# Example arithmetic: AllocCPUS=4, Elapsed=00:10:00, TotalCPU=00:20:00
#   core-seconds allocated = 4 * 600 = 2400
#   core-seconds used      = 20 * 60 = 1200
#   efficiency             = 1200 / 2400 = 50%

# Cross-check (if seff is installed)
seff <jobid>

An efficiency far below 100% on a multi-core job usually means the code is not using the cores you requested. Check the memory side the same way: `MaxRSS` against `ReqMem`. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sacct -j 42 -X --format=JobID,AllocCPUS,Elapsed,TotalCPU,State
JobID         AllocCPUS    Elapsed   TotalCPU      State 
------------ ---------- ---------- ---------- ---------- 
42                   2   00:00:04   00:00:00  COMPLETED 

$ seff 42
Job ID: 42
Cluster: slurm-training-val
User/Group: ec2-user/ec2-user
State: COMPLETED (exit code 0)
Nodes: 1
Cores per node: 2
CPU Utilized: 00:00:03
CPU Efficiency: 37.50% of 00:00:08 core-walltime
Job Wall-clock time: 00:00:04
Memory Utilized: 4.73 MB
Memory Efficiency: 0.02% of 30.40 GB (30.40 GB/node)

References

• SchedMD: Quick Start User Guide
• SchedMD: FAQ
• SchedMD: Troubleshooting Guide
• SMU Slurm Best Practices
• SchedMD: sbatch man page (--mem-update, --wait, --dependency)
• SchedMD: sbcast man page
• reportseff