Slurm Overview

<!-- audience: L1, L2 --> <!-- deployment: all --> <!-- sge-source: SGE "What is Grid Engine" + Admin Overview --> # Slurm Overview ## What is Slurm? **Slurm** (Simple Linux Utility for Resource Management) is an open-source workload manager for Linux clusters. It is the dominant job scheduler in HPC, running on many of the world's largest supercomputers and thousands of research clusters. Slurm provides three key functions: 1. **Resource allocation** -- grants users exclusive or shared access to compute nodes for a duration of time 2. **Job execution** -- provides a framework for starting, running, and monitoring work on allocated nodes 3. **Queue management** -- arbitrates contention for resources by managing a queue of pending jobs --- ## Key Concepts: Nodes and Partitions ### Nodes The physical or virtual servers in a cluster. Slurm categorizes them by role: - **Head/Login node** -- where users log in, write scripts, and submit jobs (does not run compute jobs) - **Compute nodes** -- where jobs actually execute - **Management node** -- runs the Slurm controller daemon ### Partitions Partitions are **groups of compute nodes** organized for different purposes. They replace the "queue" concept from other schedulers (SGE, PBS). Examples: | Partition | Purpose | Typical Limits | |-----------|---------|---------------| | `batch` | General-purpose compute | 7-day max walltime | | `gpu` | GPU-equipped nodes | GPU jobs only | | `highmem` | Large-memory nodes | 1TB+ RAM | | `interactive` | Short interactive sessions | 4-hour max, 1 node | | `debug` | Quick testing | 30-min max, 2 nodes | Your cluster administrators define what partitions are available. --- ## Key Concepts: Jobs, Steps, and Accounts ### Jobs A **job** is a resource allocation request plus the work to be done. Jobs can be: - **Batch jobs** -- scripts submitted with `sbatch` that run unattended - **Interactive jobs** -- real-time sessions obtained with `salloc` or `srun` - **Job arrays** -- many instances of the same job with different parameters ### Job Steps Within a job allocation, you can run one or more **steps** using `srun`. Each step can use a subset of the job's allocated resources. ### Accounts Accounts group users for **resource tracking and fairshare**. Your administrator assigns you to one or more accounts, which map to labs, departments, or projects. --- ## Architecture at a Glance Slurm runs as a set of daemons (background services): | Daemon | Runs On | Purpose | |--------|---------|---------| | `slurmctld` | Management node | Central controller -- manages jobs, nodes, partitions | | `slurmd` | Each compute node | Node daemon -- launches and monitors jobs | | `slurmdbd` | Database server | Accounting daemon -- stores job history in MySQL/MariaDB | | `slurmrestd` | API server (optional) | REST API for programmatic access | | `sackd` | Login nodes (optional) | Authentication on configless login nodes | ``` Users | v [Login Node] --submit--> [slurmctld] --dispatch--> [slurmd on compute nodes] | v [slurmdbd] --> [MySQL/MariaDB] ``` As a user, you only interact with the login node. The rest is handled transparently. --- ## Slurm vs. Other Schedulers | Feature | Slurm | SGE/UGE | PBS/Torque | LSF | |---------|-------|---------|------------|-----| | License | Open source (GPL) | Mixed | Open/Commercial | Commercial | | Scalability | Millions of cores | Thousands | Thousands | Millions | | Market position | Dominant in HPC | Legacy/declining | Common in academic | Enterprise/finance | | Cloud support | ParallelCluster, PCS, GCP | Limited | Limited | IBM Cloud | | GPU scheduling | Native GRES | Add-on | Basic | Native | Slurm has become the standard for new HPC deployments, particularly in life science, because of its open-source license, scalability, and native GPU and cloud support. --- ## Where Slurm Runs ### On-Premise Clusters Traditional bare-metal servers in a data center, connected by a high-speed network with shared storage. This is the classic HPC deployment. ### AWS ParallelCluster A self-managed, open-source tool that deploys Slurm on AWS. Key feature: **dynamic scaling** -- compute nodes launch on demand when jobs are submitted and terminate when idle, so you only pay for what you use. ### AWS PCS (Parallel Computing Service) A fully managed AWS service that provides Slurm without the operational overhead of running the controller infrastructure yourself. > **ParallelCluster Note:** ParallelCluster supports **scale-to-zero** -- compute nodes are launched on demand and terminated when idle. This means lower cost but a short startup delay when new nodes must spin up. > **PCS Note:** PCS abstracts away the Slurm control plane entirely. AWS manages the controller infrastructure, so you focus on submitting jobs rather than maintaining daemons. > **The core Slurm commands and concepts are the same across all three environments.** Deployment-specific differences are called out throughout this training. --- ## Essential Commands Preview You'll learn these commands in detail in upcoming modules: | Command | Purpose | SGE Equivalent | |---------|---------|---------------| | `sbatch` | Submit a batch job | `qsub` | | `squeue` | View job queue | `qstat` | | `scancel` | Cancel a job | `qdel` | | `sinfo` | View cluster/partition status | `qhost` | | `sacct` | View completed job info | `qacct` | | `srun` | Run interactive commands/steps | `qrsh` | | `salloc` | Request interactive allocation | `qlogin` | | `scontrol` | View/modify job and system details | `qstat -j` / `qalter` |

Exercises

1. Explore your cluster with sinfo

Run sinfo on your cluster. Identify: How many partitions are available? How many nodes are in each partition? What states are the nodes in? Try sinfo --summarize for a compact view, then sinfo --Node --long for per-node detail.

Hint / Solution

# Default summary view
sinfo

# Compact partition summary
sinfo --summarize

# Detailed per-node view
sinfo --Node --long

# Custom format showing partition, nodes, CPUs, memory, and state
sinfo --format="%12P %5a %10l %6D %8t %10C %10m"

You should see at least one partition (often called `batch` or `compute`). Node states like `idle` (available), `alloc` (fully allocated), and `mix` (partially allocated) tell you how busy the cluster is. Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      2  idle~ batch-dy-compute-[3-4]
batch*       up   infinite      2   idle batch-dy-compute-[1-2]
debug        up   infinite      2  idle~ debug-dy-small-[1-2]
gpu          up   infinite      1   idle gpu-dy-t4-1
highmem      up   infinite      1  idle~ highmem-dy-large-1

$ sinfo --summarize
PARTITION AVAIL  TIMELIMIT   NODES(A/I/O/T) NODELIST
batch*       up   infinite          0/4/0/4 batch-dy-compute-[1-4]
debug        up   infinite          0/2/0/2 debug-dy-small-[1-2]
gpu          up   infinite          0/1/0/1 gpu-dy-t4-1
highmem      up   infinite          0/1/0/1 highmem-dy-large-1

1. Check the job queue with squeue

Run squeue to see all jobs in the queue. What information does each column show? Try filtering to just your jobs with squeue --me. If the queue is empty, that is fine -- it means no jobs are running.

Hint / Solution

# View all jobs in the queue
squeue

# View only your own jobs
squeue --me

# Custom format with more detail
squeue --format="%.10i %.12j %.10u %.8T %.10M %.6D %.20R"

Key columns: `JOBID` (unique job number), `PARTITION` (which queue), `NAME` (job name), `USER`, `ST` (state: R=running, PD=pending), `TIME` (elapsed), `NODES`, and `NODELIST(REASON)` (where it is running or why it is waiting). Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ squeue
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

References

• SchedMD: Slurm Overview
• SchedMD: Quick Start User Guide
• SchedMD: Rosetta Stone of Workload Managers
• AWS ParallelCluster Slurm Overview