Architecture

# Slurm Architecture

## Overview

Slurm is a cluster workload manager that allocates resources, launches jobs, and manages queues. Its architecture is modular, scalable, and fault-tolerant -- designed to manage clusters from tens to millions of cores.

---

## Core Daemons

| Daemon | Runs On | Purpose |
|--------|---------|---------|
| **slurmctld** | Management node(s) | Central controller: job scheduling, resource allocation, state management |
| **slurmd** | Every compute node | Node daemon: launches, monitors, and manages jobs on each node |
| **slurmdbd** | Database host | Accounting daemon: stores job records, associations, and limits in MySQL/MariaDB |
| **slurmrestd** | API host (optional) | REST API daemon for programmatic Slurm access |
| **sackd** | Login nodes (optional) | Authentication daemon for configless login/submit nodes |

### Architecture Diagram

```
                    ┌─────────────────────────────────────┐
                    │          Management Node(s)          │
  Users ──────────► │  slurmctld (primary)                 │
  (login nodes)     │  slurmctld (backup, optional HA)     │
                    └──────────┬──────────────────┬────────┘
                               │                  │
                    ┌──────────▼──────┐  ┌────────▼────────┐
                    │  Compute Nodes   │  │   Database Host  │
                    │  slurmd (each)   │  │   slurmdbd       │
                    │  slurmd (each)   │  │   MySQL/MariaDB  │
                    │  slurmd (each)   │  └──────────────────┘
                    │  ...             │
                    └──────────────────┘
```

---

## slurmctld: The Central Controller

The brain of Slurm. Responsibilities:

- Maintains cluster state (nodes, partitions, jobs)
- Evaluates job requests against available resources
- Runs the scheduling algorithm (backfill by default)
- Dispatches jobs to compute nodes
- Handles job state transitions (pending → running → completed)
- Writes state checkpoints for crash recovery

**High availability:** Configure a backup slurmctld on a second host. If the primary fails, the backup takes over automatically.

```
SlurmctldHost=mgmt01(10.0.0.1)    # Primary
SlurmctldHost=mgmt02(10.0.0.2)    # Backup
```

---

## slurmd: The Node Daemon

Runs on every compute node. Responsibilities:

- Reports node resources (CPUs, memory, GPUs) to slurmctld
- Receives job launch requests from slurmctld
- Spawns job steps via `slurmstepd`
- Enforces resource limits (via cgroups)
- Reports job status and resource usage back to slurmctld

---

## slurmdbd: The Accounting Daemon

A dedicated daemon that mediates between slurmctld and the database:

- Stores job records, associations (accounts/users), fairshare data, and resource limits
- Buffers database writes for performance
- Supports multiple clusters sharing one database
- Backend: MySQL or MariaDB

**Why a separate daemon?** Decouples slurmctld from the database. If the database is slow or temporarily down, slurmctld continues to function and slurmdbd queues writes.

---

## Authentication in Slurm

Every RPC between Slurm daemons and every user command is authenticated. Without working auth, nothing runs -- jobs won't submit and daemons won't talk to each other.

Authentication is configured once in `slurm.conf` and must be identical across all nodes:

```
AuthType=auth/munge      # or auth/slurm (23.11+)
```

---

## Choosing an Auth Method

**Recommendation:** Use **Munge** for most on-prem deployments. For new clusters on Slurm 23.11+, **auth/slurm** eliminates the Munge dependency entirely.

| | **Munge** | **auth/slurm** (23.11+) |
|---|-----------|------------------------|
| How it works | Shared key + `munged` daemon on every node | Shared key file, no extra daemon |
| Key file | `/etc/munge/munge.key` | `/etc/slurm/slurm.key` |
| Maturity | Battle-tested, universal | Newer, gaining adoption |
| Best for | Existing clusters, mixed environments | New deployments, simpler stack |

**JWT** is not a primary auth method. Set it as `AuthAltType` when you need REST API access via slurmrestd.

> **ParallelCluster Note:** ParallelCluster manages Munge automatically -- do not change `AuthType`.

> **PCS Note:** Authentication is fully managed by AWS. No auth configuration is needed.

---

## Communication Model

All Slurm daemons communicate over TCP:

| Communication Path | Default Port | Protocol |
|-------------------|-------------|----------|
| Client → slurmctld | 6817 | Authenticated RPC |
| slurmctld → slurmd | 6818 | Authenticated RPC |
| slurmctld → slurmdbd | 6819 | Authenticated RPC |
| Client → slurmrestd | 6820 (configurable) | HTTPS + JWT |

---

## State Management

slurmctld maintains state in memory and periodically checkpoints to disk (`StateSaveLocation`). On restart:

- `-r` (default): Recovers jobs and DOWN/DRAIN node states
- `-R`: Full recovery including partition state and power-save settings
- `-c`: Clear all state (destructive -- rarely used in production)

---

## Plugin Architecture

Slurm's modularity comes from its plugin system. Key plugin types:

| Plugin Type | Purpose | Common Values |
|------------|---------|---------------|
| `AuthType` | Authentication | `auth/munge`, `auth/slurm` |
| `SelectType` | Resource selection | `select/cons_tres` (consumable TRES, recommended) |
| `SchedulerType` | Scheduling algorithm | `sched/backfill` (default, recommended) |
| `PriorityType` | Job prioritization | `priority/multifactor` (default) |
| `AccountingStorageType` | Accounting backend | `accounting_storage/slurmdbd` |
| `JobCompType` | Job completion logging | `jobcomp/filetxt`, `jobcomp/elasticsearch` |
| `TaskPlugin` | Task management | `task/cgroup`, `task/affinity` |
| `ProctrackType` | Process tracking | `proctrack/cgroup` |
| `GresTypes` | Generic resources | `gpu` (most common) |

---

## Deployment Variants

### On-Premise

You manage all daemons, database, and configuration. Full control, full responsibility.

### AWS ParallelCluster

> **ParallelCluster Note:** The head node runs slurmctld, slurmdbd, and MariaDB on a single EC2 instance. There is no native HA for the controller -- if the head node is lost, it must be recreated. Size the head node instance appropriately (e.g., `c5.2xlarge` or larger for clusters >500 nodes).

ParallelCluster manages slurmctld, slurmdbd, and the database on the head node. It adds:

- Dynamic compute nodes that launch on demand (EC2 instances)
- Power-saving daemon that terminates idle nodes
- Custom slurm.conf parameters via `CustomSlurmSettings`
- Some slurm.conf parameters are deny-listed (managed by ParallelCluster)

### AWS PCS

> **PCS Note:** The Slurm control plane (slurmctld, slurmdbd, database) is fully managed by AWS and does not run in your account. You interact with it through the PCS API and console -- there is no SSH access to the controller.

A fully managed Slurm control plane. You configure through the PCS API rather than editing slurm.conf directly.

Exercises¶

Identify running Slurm daemons

Log into your cluster's head node and compute node. Use systemctl or ps to identify which Slurm daemons are running on each. Determine which node runs slurmctld, which runs slurmd, and where slurmdbd is running (if applicable).

Hint / Solution

# On the head node — check for the controller and database daemons
systemctl status slurmctld
systemctl status slurmdbd

# On a compute node — check for the node daemon
systemctl status slurmd

# Alternative: use ps to find all Slurm processes on any node
ps aux | grep -E 'slurm(ctld|dbd|d|restd)' | grep -v grep

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ systemctl is-active slurmctld slurmdbd
active
active

$ ps -eo comm,pid | grep -E 'slurm(ctld|dbd|d|restd)' | head -10
slurmdbd         180649
slurmctld        180673

Check daemon status and version

Verify the Slurm version running on the cluster using two different methods. Then use scontrol to confirm the controller is responding and display key configuration parameters.

Hint / Solution

# Method 1: Direct daemon version query
slurmctld -V
slurmd -V

# Method 2: From the running configuration
scontrol show config | grep SlurmVersion

# Verify the controller is responding
scontrol ping

# Show key configuration parameters
scontrol show config | grep -E '(ClusterName|SlurmctldHost|AccountingStorageType)'

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ slurmctld -V
slurm 25.11.4

$ scontrol show config | grep SLURM_VERSION
SLURM_VERSION           = 25.11.4

$ scontrol ping
Slurmctld(primary) at ip-172-31-94-160 is UP

Architecture

Exercises¶

References¶