Partitions & QOS

# Partitions & Quality of Service (QOS)

## Partitions: The Basics

Partitions are Slurm's equivalent of SGE "cluster queues." They group compute nodes and define scheduling boundaries with their own limits, access controls, and defaults.

A node can belong to multiple partitions. A job targets one partition.

### Partition Configuration

> **ParallelCluster Note:** Partitions are auto-generated from `SlurmQueues` in your cluster YAML config. You cannot modify partition definitions in slurm.conf directly -- changes are overwritten on updates. Use `CustomSlurmSettings` for tuning partition properties like `DefaultTime` or `PriorityTier`. QOS can be added normally via `sacctmgr`.

> **PCS Note:** Queues (partitions) map to compute node groups defined through the PCS API. Partition properties like `MaxTime` and `DefaultTime` are set at the queue level in the PCS console or CLI.

```ini
# slurm.conf
PartitionName=batch    Nodes=node[001-048]  Default=YES  MaxTime=7-00:00:00  DefaultTime=01:00:00  State=UP
PartitionName=gpu      Nodes=gpu[01-08]     MaxTime=2-00:00:00  DefaultTime=04:00:00  State=UP
PartitionName=highmem  Nodes=bigmem[1-4]    MaxTime=7-00:00:00  State=UP
PartitionName=debug    Nodes=node[001-004]  MaxTime=00:30:00  MaxNodes=2  PriorityTier=100  State=UP
```

---

## Key Partition Properties

| Property | Purpose | Example |
|----------|---------|---------|
| `Default=YES` | Used when user doesn't specify `-p` | One partition only |
| `MaxTime` | Absolute maximum walltime | `7-00:00:00` (7 days) |
| `DefaultTime` | Default if user omits `--time` | `01:00:00` |
| `MaxNodes` | Max nodes per job | `16` |
| `MaxCPUsPerNode` | Limit CPUs per node in this partition | `32` |
| `PriorityTier` | Evaluation order (higher = evaluated first) | `100` |
| `PriorityJobFactor` | Contribution to multifactor priority | `10` |
| `AllowGroups` | Unix group access control | `biolab,physics` |
| `AllowAccounts` | Slurm account access control | `labA,labB` |
| `AllowQos` | Which QOS can target this partition | `normal,high` |
| `QOS` | Attach a partition-level QOS | `partition_gpu_qos` |
| `OverSubscribe` | Node sharing policy | `YES`, `NO`, `FORCE` |
| `PreemptMode` | Preemption behavior | `REQUEUE`, `CANCEL` |
| `State` | Partition availability | `UP`, `DOWN`, `DRAIN`, `INACTIVE` |

---

## Partition Design Patterns

**By hardware type:**
```ini
PartitionName=cpu     Nodes=node[001-100]   MaxTime=7-00:00:00
PartitionName=gpu     Nodes=gpu[01-16]      MaxTime=2-00:00:00
PartitionName=highmem Nodes=bigmem[1-8]     MaxTime=7-00:00:00
```

**By priority/turnaround:**
```ini
PartitionName=debug       MaxTime=00:30:00  MaxNodes=2   PriorityTier=200
PartitionName=interactive MaxTime=04:00:00  MaxNodes=1   PriorityTier=150
PartitionName=normal      MaxTime=2-00:00:00              PriorityTier=100  Default=YES
PartitionName=long        MaxTime=14-00:00:00             PriorityTier=50
```

**By research group (overlapping nodes):**
```ini
PartitionName=all     Nodes=node[001-100]  Default=YES
PartitionName=labA    Nodes=node[001-025]  AllowAccounts=labA
PartitionName=labB    Nodes=node[026-050]  AllowAccounts=labB
```

---

## QOS: Quality of Service

QOS provides **finer-grained control** than partitions alone. While partitions define *where* jobs can run, QOS defines *how much* and *at what priority*.

QOS is managed through `sacctmgr` (stored in the Slurm database), not in slurm.conf.

### Creating QOS

```bash
# Create QOS tiers
sacctmgr add qos normal
sacctmgr add qos high     set Priority=1000 MaxTRESPerUser=cpu=256
sacctmgr add qos low      set Priority=-500 MaxTRESPerUser=cpu=512 GrpTRES=cpu=1024
sacctmgr add qos unlimited set Priority=500 MaxTRES=cpu=2000
```

### QOS Limits

```bash
sacctmgr modify qos normal set \
    MaxTRESPerUser=cpu=128,mem=512G,gres/gpu=4 \
    MaxJobsPerUser=50 \
    MaxSubmitJobsPerUser=500 \
    MaxWallDurationPerJob=7-00:00:00 \
    GrpTRES=cpu=2048
```

Key QOS limit fields:

| Limit | Scope | Description |
|-------|-------|-------------|
| `GrpTRES` | All jobs in QOS | Total TRES across all running jobs |
| `GrpJobs` | All jobs in QOS | Max concurrent running jobs |
| `MaxTRESPerUser` | Per user | Max TRES per user across running jobs |
| `MaxTRESPerJob` | Per job | Max TRES for a single job |
| `MaxJobsPerUser` | Per user | Max running jobs per user |
| `MaxSubmitJobsPerUser` | Per user | Max submitted (pending + running) per user |
| `MaxWallDurationPerJob` | Per job | Maximum walltime |
| `MinTRESPerJob` | Per job | Minimum TRES (ensures minimum allocation) |

---

## QOS Priority and Preemption

### QOS Priority

QOS contributes to the multifactor priority calculation via:

```bash
sacctmgr modify qos high set Priority=1000
```

This adds 1000 * `PriorityWeightQOS` to jobs using this QOS.

### QOS Preemption

QOS can define preemption relationships:

```bash
# high QOS can preempt normal and low
sacctmgr modify qos high set PreemptMode=REQUEUE Preempt=normal,low

# Set grace time before preemption
sacctmgr modify qos normal set GraceTime=300    # 5-minute warning
```

### Assigning QOS to Users

```bash
# Allow user to use specific QOS
sacctmgr modify user jdoe set qos=normal,high
sacctmgr modify user jdoe set defaultqos=normal

# Allow an entire account to use a QOS
sacctmgr modify account labA set qos+=high
```

Users request a QOS at submission:

```bash
sbatch --qos=high my_job.sh
```

---

## Partition QOS

Attach a QOS to a partition to impose floating resource limits:

```ini
# slurm.conf
PartitionName=gpu Nodes=gpu[01-16] QOS=gpu_partition_qos MaxTime=2-00:00:00
```

```bash
# Create the partition QOS
sacctmgr add qos gpu_partition_qos set GrpTRES=gres/gpu=32 MaxTRESPerUser=gres/gpu=8
```

This means: the GPU partition as a whole can use 32 GPUs total, and no single user can use more than 8. The actual job QOS (requested by the user) determines priority and preemption; the partition QOS just sets overall limits.

### Relative QOS (Percentage-Based)

New in recent Slurm versions -- set limits as percentages:

```bash
sacctmgr add qos dept_limit set Flags=Relative GrpTRES=cpu=25
# This QOS can use 25% of the cluster's (or partition's) CPUs
```

---

## Putting It Together: A Real Example

**Scenario:** A life science cluster with 100 CPU nodes, 8 GPU nodes, 4 high-memory nodes.

```ini
# slurm.conf - Partitions
PartitionName=batch     Nodes=node[001-100]  Default=YES  MaxTime=7-00:00:00  DefaultTime=01:00:00
PartitionName=gpu       Nodes=gpu[01-08]     MaxTime=2-00:00:00  DefaultTime=04:00:00  QOS=gpu_qos
PartitionName=highmem   Nodes=bigmem[1-4]    MaxTime=7-00:00:00
PartitionName=debug     Nodes=node[001-004]  MaxTime=00:30:00  MaxNodes=2  PriorityTier=200
```

```bash
# QOS setup
sacctmgr add qos normal    set Priority=0    MaxJobsPerUser=100  MaxTRESPerUser=cpu=256
sacctmgr add qos high      set Priority=1000 MaxJobsPerUser=200  MaxTRESPerUser=cpu=512  Preempt=low
sacctmgr add qos low       set Priority=-500 MaxJobsPerUser=50   MaxTRESPerUser=cpu=1024
sacctmgr add qos gpu_qos   set GrpTRES=gres/gpu=24 MaxTRESPerUser=gres/gpu=4
```

Exercises¶

Create a new partition for long-running jobs

Add a long partition in slurm.conf that uses nodes cpu[001-064], allows a maximum walltime of 14 days, and has a lower PriorityTier than the default batch partition. Apply the change and verify the partition exists.

Hint / Solution

# Add to slurm.conf:
#   PartitionName=long Nodes=cpu[001-064] MaxTime=14-00:00:00 DefaultTime=04:00:00 PriorityTier=50 State=UP

# Apply
scontrol reconfigure

# Verify
scontrol show partition long
sinfo -p long

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ scontrol show partition val_long
PartitionName=val_long
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=04:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=14-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   Nodes=debug-dy-small-[1-2]
   PriorityJobFactor=1 PriorityTier=50 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=4 TotalNodes=2 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=4,mem=7782M,node=2,billing=4
   ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO

$ sinfo -p val_long
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
val_long     up 14-00:00:0      2  idle~ debug-dy-small-[1-2]

Create a QOS with specific resource limits

Create a QOS called restricted that limits each user to 32 CPUs and 2 GPUs across all their running jobs, and caps concurrent running jobs at 10 per user.

Hint / Solution

sacctmgr add qos restricted set \
    MaxTRESPerUser=cpu=32,gres/gpu=2 \
    MaxJobsPerUser=10 \
    Priority=0

# Verify
sacctmgr show qos restricted format=Name,MaxTRESPerUser,MaxJobsPerUser,Priority

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sudo /opt/slurm/bin/sacctmgr -i add qos val_pq_restricted set          MaxTRESPerUser=cpu=32          MaxJobsPerUser=10          Priority=0
 Adding QOS(s)
  val_pq_restricted
 Settings
  Description    = val_pq_restricted
  MaxJobsPerUser = 10
  MaxTRESPerUser           = cpu=32
  Priority                 = 0

$ sacctmgr show qos val_pq_restricted format=Name%-20,MaxTRESPerUser%-30,MaxJobsPerUser,Priority
Name                 MaxTRESPU                      MaxJobsPU   Priority 
-------------------- ------------------------------ --------- ---------- 
val_pq_restricted    cpu=32                                10          0

Assign a QOS to an account

Allow all users in the smith_lab account to use both the normal and restricted QOS. Set normal as their default.

Hint / Solution

# Add the QOS to the account
sacctmgr modify account smith_lab set qos=normal,restricted defaultqos=normal

# Verify
sacctmgr show association where account=smith_lab format=Account,User,QOS,DefaultQOS

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sudo /opt/slurm/bin/sacctmgr -i modify account training set qos=normal,val_pq_restricted defaultqos=normal
 Modified account associations...
  C = slurm-training-val A = training of root

$ sacctmgr show association where account=training format=Account,User,QOS,DefaultQOS
   Account       User                  QOS   Def QOS 
---------- ---------- -------------------- --------- 
  training            normal,val_pq_restr+    normal

Test that a QOS limit prevents over-allocation

Create a test QOS called tiny that allows a maximum of 2 CPUs per user total. Assign it to a test user. Submit two single-CPU jobs (they should start), then submit a third and confirm it is held pending with a QOS limit reason.

Hint / Solution

# Create the restrictive QOS
sacctmgr add qos tiny set MaxTRESPerUser=cpu=2 MaxJobsPerUser=10

# Assign to a test user
sacctmgr modify user testuser set qos=tiny defaultqos=tiny

# As testuser, submit jobs:
su - testuser
sbatch --qos=tiny --wrap="sleep 300" -n 1
sbatch --qos=tiny --wrap="sleep 300" -n 1
sbatch --qos=tiny --wrap="sleep 300" -n 1

# Check status -- the third job should be pending
squeue -u testuser -o "%.8i %.2t %.20R"
# Expected reason: (QOSMaxCpuPerUserLimit) or (QOSMaxTRESPerUser)

# Clean up
scancel -u testuser
sacctmgr delete qos tiny

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ squeue -u testuser1 -o '%.8i %.2t %.30R'
   JOBID ST               NODELIST(REASON)
     617 PD        (QOSMaxCpuPerUserLimit)
     616  R             batch-dy-compute-2
     615  R             batch-dy-compute-1

Partitions & QOS

Exercises¶

References¶