Partitions & QOS

# Partitions & Quality of Service (QOS)

## Partitions: The Basics

Partitions are Slurm's equivalent of SGE "cluster queues." They group compute nodes and define scheduling boundaries with their own limits, access controls, and defaults.

A node can belong to multiple partitions. A job targets one partition.

### Partition Configuration

> **ParallelCluster Note:** Partitions are auto-generated from `SlurmQueues` in your cluster YAML config. You cannot modify partition definitions in slurm.conf directly -- changes are overwritten on updates. Use `CustomSlurmSettings` for tuning partition properties like `DefaultTime` or `PriorityTier`. QOS can be added normally via `sacctmgr`.

> **PCS Note:** Queues (partitions) map to compute node groups defined through the PCS API. Partition properties like `MaxTime` and `DefaultTime` are set at the queue level in the PCS console or CLI.

```ini
# slurm.conf
PartitionName=batch    Nodes=node[001-048]  Default=YES  MaxTime=7-00:00:00  DefaultTime=01:00:00  State=UP
PartitionName=gpu      Nodes=gpu[01-08]     MaxTime=2-00:00:00  DefaultTime=04:00:00  State=UP
PartitionName=highmem  Nodes=bigmem[1-4]    MaxTime=7-00:00:00  State=UP
PartitionName=debug    Nodes=node[001-004]  MaxTime=00:30:00  MaxNodes=2  PriorityTier=100  State=UP
```

### Key Partition Properties

| Property | Purpose | Example |
|----------|---------|---------|
| `Default=YES` | Used when user doesn't specify `-p` | One partition only |
| `MaxTime` | Absolute maximum walltime | `7-00:00:00` (7 days) |
| `DefaultTime` | Default if user omits `--time` | `01:00:00` |
| `MaxNodes` | Max nodes per job | `16` |
| `MaxCPUsPerNode` | Limit CPUs per node in this partition | `32` |
| `PriorityTier` | Evaluation order (higher = evaluated first) | `100` |
| `PriorityJobFactor` | Contribution to multifactor priority | `10` |
| `AllowGroups` | Unix group access control | `biolab,physics` |
| `AllowAccounts` | Slurm account access control | `labA,labB` |
| `AllowQos` | Which QOS can target this partition | `normal,high` |
| `QOS` | Attach a partition-level QOS | `partition_gpu_qos` |
| `OverSubscribe` | Node sharing policy | `YES`, `NO`, `FORCE` |
| `PreemptMode` | Preemption behavior | `REQUEUE`, `CANCEL` |
| `State` | Partition availability | `UP`, `DOWN`, `DRAIN`, `INACTIVE` |

### Design Patterns

**By hardware type:**
```ini
PartitionName=cpu     Nodes=node[001-100]   MaxTime=7-00:00:00
PartitionName=gpu     Nodes=gpu[01-16]      MaxTime=2-00:00:00
PartitionName=highmem Nodes=bigmem[1-8]     MaxTime=7-00:00:00
```

**By priority/turnaround:**
```ini
PartitionName=debug       MaxTime=00:30:00  MaxNodes=2   PriorityTier=200
PartitionName=interactive MaxTime=04:00:00  MaxNodes=1   PriorityTier=150
PartitionName=normal      MaxTime=2-00:00:00              PriorityTier=100  Default=YES
PartitionName=long        MaxTime=14-00:00:00             PriorityTier=50
```

**By research group (overlapping nodes):**
```ini
PartitionName=all     Nodes=node[001-100]  Default=YES
PartitionName=labA    Nodes=node[001-025]  AllowAccounts=labA
PartitionName=labB    Nodes=node[026-050]  AllowAccounts=labB
```

---

## QOS: Quality of Service

QOS provides **finer-grained control** than partitions alone. While partitions define *where* jobs can run, QOS defines *how much* and *at what priority*.

QOS is managed through `sacctmgr` (stored in the Slurm database), not in slurm.conf.

### Creating QOS

```bash
# Create QOS tiers
sacctmgr add qos normal
sacctmgr add qos high     set Priority=1000 MaxTRESPerUser=cpu=256
sacctmgr add qos low      set Priority=-500 MaxTRESPerUser=cpu=512 GrpTRES=cpu=1024
sacctmgr add qos unlimited set Priority=500 MaxTRES=cpu=2000
```

### QOS Limits

```bash
sacctmgr modify qos normal set \
    MaxTRESPerUser=cpu=128,mem=512G,gres/gpu=4 \
    MaxJobsPerUser=50 \
    MaxSubmitJobsPerUser=500 \
    MaxWallDurationPerJob=7-00:00:00 \
    GrpTRES=cpu=2048
```

Key QOS limit fields:

| Limit | Scope | Description |
|-------|-------|-------------|
| `GrpTRES` | All jobs in QOS | Total TRES across all running jobs |
| `GrpJobs` | All jobs in QOS | Max concurrent running jobs |
| `MaxTRESPerUser` | Per user | Max TRES per user across running jobs |
| `MaxTRESPerJob` | Per job | Max TRES for a single job |
| `MaxJobsPerUser` | Per user | Max running jobs per user |
| `MaxSubmitJobsPerUser` | Per user | Max submitted (pending + running) per user |
| `MaxWallDurationPerJob` | Per job | Maximum walltime |
| `MinTRESPerJob` | Per job | Minimum TRES (ensures minimum allocation) |

### QOS Priority

QOS contributes to the multifactor priority calculation via:

```bash
sacctmgr modify qos high set Priority=1000
```

This adds 1000 * `PriorityWeightQOS` to jobs using this QOS.

### QOS Preemption

QOS can define preemption relationships:

```bash
# high QOS can preempt normal and low
sacctmgr modify qos high set PreemptMode=REQUEUE Preempt=normal,low

# Set grace time before preemption
sacctmgr modify qos normal set GraceTime=300    # 5-minute warning
```

### Assigning QOS to Users

```bash
# Allow user to use specific QOS
sacctmgr modify user jdoe set qos=normal,high
sacctmgr modify user jdoe set defaultqos=normal

# Allow an entire account to use a QOS
sacctmgr modify account labA set qos+=high
```

Users request a QOS at submission:

```bash
sbatch --qos=high my_job.sh
```

---

## Partition QOS

Attach a QOS to a partition to impose floating resource limits:

```ini
# slurm.conf
PartitionName=gpu Nodes=gpu[01-16] QOS=gpu_partition_qos MaxTime=2-00:00:00
```

```bash
# Create the partition QOS
sacctmgr add qos gpu_partition_qos set GrpTRES=gres/gpu=32 MaxTRESPerUser=gres/gpu=8
```

This means: the GPU partition as a whole can use 32 GPUs total, and no single user can use more than 8. The actual job QOS (requested by the user) determines priority and preemption; the partition QOS just sets overall limits.

### Relative QOS (Percentage-Based)

New in recent Slurm versions -- set limits as percentages:

```bash
sacctmgr add qos dept_limit set Flags=Relative GrpTRES=cpu=25
# This QOS can use 25% of the cluster's (or partition's) CPUs
```

---

## Putting It Together: A Real Example

**Scenario:** A life science cluster with 100 CPU nodes, 8 GPU nodes, 4 high-memory nodes.

```ini
# slurm.conf - Partitions
PartitionName=batch     Nodes=node[001-100]  Default=YES  MaxTime=7-00:00:00  DefaultTime=01:00:00
PartitionName=gpu       Nodes=gpu[01-08]     MaxTime=2-00:00:00  DefaultTime=04:00:00  QOS=gpu_qos
PartitionName=highmem   Nodes=bigmem[1-4]    MaxTime=7-00:00:00
PartitionName=debug     Nodes=node[001-004]  MaxTime=00:30:00  MaxNodes=2  PriorityTier=200
```

```bash
# QOS setup
sacctmgr add qos normal    set Priority=0    MaxJobsPerUser=100  MaxTRESPerUser=cpu=256
sacctmgr add qos high      set Priority=1000 MaxJobsPerUser=200  MaxTRESPerUser=cpu=512  Preempt=low
sacctmgr add qos low       set Priority=-500 MaxJobsPerUser=50   MaxTRESPerUser=cpu=1024
sacctmgr add qos gpu_qos   set GrpTRES=gres/gpu=24 MaxTRESPerUser=gres/gpu=4
```

Exercises¶

Create a new partition for long-running jobs

Add a long partition in slurm.conf that uses nodes cpu[001-064], allows a maximum walltime of 14 days, and has a lower PriorityTier than the default batch partition. Apply the change and verify the partition exists.

Hint / Solution

# Add to slurm.conf:
#   PartitionName=long Nodes=cpu[001-064] MaxTime=14-00:00:00 DefaultTime=04:00:00 PriorityTier=50 State=UP

# Apply
scontrol reconfigure

# Verify
scontrol show partition long
sinfo -p long

Create a QOS with specific resource limits

Create a QOS called restricted that limits each user to 32 CPUs and 2 GPUs across all their running jobs, and caps concurrent running jobs at 10 per user.

Hint / Solution

sacctmgr add qos restricted set \
    MaxTRESPerUser=cpu=32,gres/gpu=2 \
    MaxJobsPerUser=10 \
    Priority=0

# Verify
sacctmgr show qos restricted format=Name,MaxTRESPerUser,MaxJobsPerUser,Priority

Assign a QOS to an account

Allow all users in the smith_lab account to use both the normal and restricted QOS. Set normal as their default.

Hint / Solution

# Add the QOS to the account
sacctmgr modify account smith_lab set qos=normal,restricted defaultqos=normal

# Verify
sacctmgr show association where account=smith_lab format=Account,User,QOS,DefaultQOS

Test that a QOS limit prevents over-allocation

Create a test QOS called tiny that allows a maximum of 2 CPUs per user total. Assign it to a test user. Submit two single-CPU jobs (they should start), then submit a third and confirm it is held pending with a QOS limit reason.

Hint / Solution

# Create the restrictive QOS
sacctmgr add qos tiny set MaxTRESPerUser=cpu=2 MaxJobsPerUser=10

# Assign to a test user
sacctmgr modify user testuser set qos=tiny defaultqos=tiny

# As testuser, submit jobs:
su - testuser
sbatch --qos=tiny --wrap="sleep 300" -n 1
sbatch --qos=tiny --wrap="sleep 300" -n 1
sbatch --qos=tiny --wrap="sleep 300" -n 1

# Check status -- the third job should be pending
squeue -u testuser -o "%.8i %.2t %.20R"
# Expected reason: (QOSMaxCpuPerUserLimit) or (QOSMaxTRESPerUser)

# Clean up
scancel -u testuser
sacctmgr delete qos tiny

Partitions & QOS

Exercises¶

References¶