Partitions & QOS

<!-- audience: L4 --> <!-- deployment: all --> <!-- sge-source: SGE queues + queue configuration --> # Partitions & Quality of Service (QOS) ## Partitions: The Basics Partitions are Slurm's equivalent of SGE "cluster queues." They group compute nodes and define scheduling boundaries with their own limits, access controls, and defaults. A node can belong to multiple partitions. A job targets one partition. ### Partition Configuration > **ParallelCluster Note:** Partitions are auto-generated from `SlurmQueues` in your cluster YAML config. You cannot modify partition definitions in slurm.conf directly -- changes are overwritten on updates. Use `CustomSlurmSettings` for tuning partition properties like `DefaultTime` or `PriorityTier`. QOS can be added normally via `sacctmgr`. > **PCS Note:** Queues (partitions) map to compute node groups defined through the PCS API. Partition properties like `MaxTime` and `DefaultTime` are set at the queue level in the PCS console or CLI. ```ini # slurm.conf PartitionName=batch Nodes=node[001-048] Default=YES MaxTime=7-00:00:00 DefaultTime=01:00:00 State=UP PartitionName=gpu Nodes=gpu[01-08] MaxTime=2-00:00:00 DefaultTime=04:00:00 State=UP PartitionName=highmem Nodes=bigmem[1-4] MaxTime=7-00:00:00 State=UP PartitionName=debug Nodes=node[001-004] MaxTime=00:30:00 MaxNodes=2 PriorityTier=100 State=UP ``` --- ## Key Partition Properties | Property | Purpose | Example | |----------|---------|---------| | `Default=YES` | Used when user doesn't specify `-p` | One partition only | | `MaxTime` | Absolute maximum walltime | `7-00:00:00` (7 days) | | `DefaultTime` | Default if user omits `--time` | `01:00:00` | | `MaxNodes` | Max nodes per job | `16` | | `MaxCPUsPerNode` | Limit CPUs per node in this partition | `32` | | `PriorityTier` | Evaluation order (higher = evaluated first) | `100` | | `PriorityJobFactor` | Contribution to multifactor priority | `10` | | `AllowGroups` | Unix group access control | `biolab,physics` | | `AllowAccounts` | Slurm account access control | `labA,labB` | | `AllowQos` | Which QOS can target this partition | `normal,high` | | `QOS` | Attach a partition-level QOS | `partition_gpu_qos` | | `OverSubscribe` | Node sharing policy | `YES`, `NO`, `FORCE` | | `PreemptMode` | Preemption behavior | `REQUEUE`, `CANCEL` | | `State` | Partition availability | `UP`, `DOWN`, `DRAIN`, `INACTIVE` | > **Hidden partitions:** `sinfo`, `squeue` and `scontrol show node` omit `Hidden=YES` partitions, and partitions the caller's group may not use, unless you pass `-a`. A node census without `-a` under-counts the fleet. Use `scontrol -a show partition | grep -E 'PartitionName|Hidden|AllowGroups|AllowQos'`. **26.05+:** `Exclusive=[NO|NODE|USER|TOPO]` replaces `ExclusiveUser`, `ExclusiveTopo` and `OverSubscribe=EXCLUSIVE`. `NODE` allocates whole nodes, `USER` allows one user's jobs per node, `TOPO` allows one job per topology segment. The old keys are still accepted. --- ## Partition Policy Pitfalls **`AllowGroups=` with few members strands cores.** Quantify first: allocated core-hours in the owner partition vs capacity over a month (`sacct -a -X -P -n -o Partition,AllocCPUS,Elapsed`), against pending core-hours in the general queue. If the owner partition is mostly idle, use a preemptable owner partition (lower-priority general partition over the same nodes, `PreemptType=preempt/partition_prio`) or `OverSubscribe`. Preemption needs requeue-safe work and a non-FIFO scheduler. **`MaxTime=UNLIMITED` means no reaper.** A hung job holds its cores until a human cancels it. A finite `MaxTime` turns hung jobs into `TIMEOUT` records you can count. **Minimal-policy signature:** ```ini SchedulerType=sched/builtin PriorityType=priority/basic PreemptMode=OFF MaxTime=UNLIMITED # every partition SelectTypeParameters=CR_CPU # DefMemPerNode=UNLIMITED ``` That reads as strict FIFO: no fair-share, no backfill, memory unscheduled. It is a common vendor default, so report it as a design, not a misconfiguration. --- ## Partition Design Patterns **By hardware type:** ```ini PartitionName=cpu Nodes=node[001-100] MaxTime=7-00:00:00 PartitionName=gpu Nodes=gpu[01-16] MaxTime=2-00:00:00 PartitionName=highmem Nodes=bigmem[1-8] MaxTime=7-00:00:00 ``` **By priority/turnaround:** ```ini PartitionName=debug MaxTime=00:30:00 MaxNodes=2 PriorityTier=200 PartitionName=interactive MaxTime=04:00:00 MaxNodes=1 PriorityTier=150 PartitionName=normal MaxTime=2-00:00:00 PriorityTier=100 Default=YES PartitionName=long MaxTime=14-00:00:00 PriorityTier=50 ``` **By research group (overlapping nodes):** ```ini PartitionName=all Nodes=node[001-100] Default=YES PartitionName=labA Nodes=node[001-025] AllowAccounts=labA PartitionName=labB Nodes=node[026-050] AllowAccounts=labB ``` --- ## QOS: Quality of Service QOS provides **finer-grained control** than partitions alone. While partitions define *where* jobs can run, QOS defines *how much* and *at what priority*. QOS is managed through `sacctmgr` (stored in the Slurm database), not in slurm.conf. ### Creating QOS ```bash # Create QOS tiers sacctmgr add qos normal sacctmgr add qos high set Priority=1000 MaxTRESPerUser=cpu=256 sacctmgr add qos low set Priority=-500 MaxTRESPerUser=cpu=512 GrpTRES=cpu=1024 sacctmgr add qos unlimited set Priority=500 MaxTRES=cpu=2000 ``` ### QOS Limits ```bash sacctmgr modify qos normal set \ MaxTRESPerUser=cpu=128,mem=512G,gres/gpu=4 \ MaxJobsPerUser=50 \ MaxSubmitJobsPerUser=500 \ MaxWallDurationPerJob=7-00:00:00 \ GrpTRES=cpu=2048 ``` Key QOS limit fields: | Limit | Scope | Description | |-------|-------|-------------| | `GrpTRES` | All jobs in QOS | Total TRES across all running jobs | | `GrpJobs` | All jobs in QOS | Max concurrent running jobs | | `MaxTRESPerUser` | Per user | Max TRES per user across running jobs | | `MaxTRESPerJob` | Per job | Max TRES for a single job | | `MaxJobsPerUser` | Per user | Max running jobs per user | | `MaxSubmitJobsPerUser` | Per user | Max submitted (pending + running) per user | | `MaxWallDurationPerJob` | Per job | Maximum walltime | | `MinTRESPerJob` | Per job | Minimum TRES (ensures minimum allocation) | --- ## QOS Priority and Preemption ### QOS Priority QOS contributes to the multifactor priority calculation via: ```bash sacctmgr modify qos high set Priority=1000 ``` This adds 1000 * `PriorityWeightQOS` to jobs using this QOS. > **Queue-jumping warning:** if the QOS rows differ only by `Priority`, partitions keep the default `AllowQos=ALL`, associations list every QOS, and `PriorityWeightQOS` is orders of magnitude above `PriorityWeightFairShare`, anyone can run `sbatch --qos=high` and jump the queue. Fair-share is then effectively off. Restrict who may use the high QOS through the association `QosLevel` (`sacctmgr modify user <user> set qos=normal`), and check with `sshare -a -l`: FairShare varies while pending jobs still order by QOS. ### QOS Preemption QOS can define preemption relationships: ```bash # high QOS can preempt normal and low sacctmgr modify qos high set PreemptMode=REQUEUE Preempt=normal,low # Set grace time before preemption sacctmgr modify qos normal set GraceTime=300 # 5-minute warning ``` ### Assigning QOS to Users ```bash # Allow user to use specific QOS sacctmgr modify user jdoe set qos=normal,high sacctmgr modify user jdoe set defaultqos=normal # Allow an entire account to use a QOS sacctmgr modify account labA set qos+=high ``` Users request a QOS at submission: ```bash sbatch --qos=high my_job.sh ``` --- ## Partition QOS Attach a QOS to a partition to impose floating resource limits: ```ini # slurm.conf PartitionName=gpu Nodes=gpu[01-16] QOS=gpu_partition_qos MaxTime=2-00:00:00 ``` ```bash # Create the partition QOS sacctmgr add qos gpu_partition_qos set GrpTRES=gres/gpu=32 MaxTRESPerUser=gres/gpu=8 ``` This means: the GPU partition as a whole can use 32 GPUs total, and no single user can use more than 8. The actual job QOS (requested by the user) determines priority and preemption; the partition QOS just sets overall limits. ### Relative QOS (Percentage-Based) New in recent Slurm versions -- set limits as percentages: ```bash sacctmgr add qos dept_limit set Flags=Relative GrpTRES=cpu=25 # This QOS can use 25% of the cluster's (or partition's) CPUs ``` --- ## Putting It Together: A Real Example **Scenario:** A life science cluster with 100 CPU nodes, 8 GPU nodes, 4 high-memory nodes. ```ini # slurm.conf - Partitions PartitionName=batch Nodes=node[001-100] Default=YES MaxTime=7-00:00:00 DefaultTime=01:00:00 PartitionName=gpu Nodes=gpu[01-08] MaxTime=2-00:00:00 DefaultTime=04:00:00 QOS=gpu_qos PartitionName=highmem Nodes=bigmem[1-4] MaxTime=7-00:00:00 PartitionName=debug Nodes=node[001-004] MaxTime=00:30:00 MaxNodes=2 PriorityTier=200 ``` ```bash # QOS setup sacctmgr add qos normal set Priority=0 MaxJobsPerUser=100 MaxTRESPerUser=cpu=256 sacctmgr add qos high set Priority=1000 MaxJobsPerUser=200 MaxTRESPerUser=cpu=512 Preempt=low sacctmgr add qos low set Priority=-500 MaxJobsPerUser=50 MaxTRESPerUser=cpu=1024 sacctmgr add qos gpu_qos set GrpTRES=gres/gpu=24 MaxTRESPerUser=gres/gpu=4 ```

Exercises

1. Create a new partition for long-running jobs

Add a long partition in slurm.conf that uses nodes cpu[001-064], allows a maximum walltime of 14 days, and has a lower PriorityTier than the default batch partition. Apply the change and verify the partition exists.

Hint / Solution

# Add to slurm.conf:
#   PartitionName=long Nodes=cpu[001-064] MaxTime=14-00:00:00 DefaultTime=04:00:00 PriorityTier=50 State=UP

# Apply
scontrol reconfigure

# Verify
scontrol show partition long
sinfo -p long

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ scontrol show partition val_long
PartitionName=val_long
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=04:00:00 DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=14-00:00:00 MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   Nodes=debug-dy-small-[1-2]
   PriorityJobFactor=1 PriorityTier=50 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=4 TotalNodes=2 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=4,mem=7782M,node=2,billing=4
   ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO

$ sinfo -p val_long
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
val_long     up 14-00:00:0      2  idle~ debug-dy-small-[1-2]

1. Create a QOS with specific resource limits

Create a QOS called restricted that limits each user to 32 CPUs and 2 GPUs across all their running jobs, and caps concurrent running jobs at 10 per user.

Hint / Solution

sacctmgr add qos restricted set \
    MaxTRESPerUser=cpu=32,gres/gpu=2 \
    MaxJobsPerUser=10 \
    Priority=0

# Verify
sacctmgr show qos restricted format=Name,MaxTRESPerUser,MaxJobsPerUser,Priority

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

# exit=1
$ sudo /opt/slurm/bin/sacctmgr -i add qos val_pq_restricted set          MaxTRESPerUser=cpu=32,gres/gpu=2          MaxJobsPerUser=10          Priority=0
 Nothing new added.

$ sacctmgr show qos val_pq_restricted format=Name%-20,MaxTRESPerUser%-30,MaxJobsPerUser,Priority
Name                 MaxTRESPU                      MaxJobsPU   Priority 
-------------------- ------------------------------ --------- ---------- 
val_pq_restricted    cpu=32,gres/gpu=2                     10          0 

1. Assign a QOS to an account

Allow all users in the smith_lab account to use both the normal and restricted QOS. Set normal as their default.

Hint / Solution

# Add the QOS to the account
sacctmgr modify account smith_lab set qos=normal,restricted defaultqos=normal

# Verify
sacctmgr show association where account=smith_lab format=Account,User,QOS,DefaultQOS

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sudo /opt/slurm/bin/sacctmgr -i modify account training set qos=normal,val_pq_restricted defaultqos=normal
 Modified account associations...
  C = slurm-training-val A = training of root

$ sacctmgr show association where account=training format=Account,User,QOS,DefaultQOS
   Account       User                  QOS   Def QOS 
---------- ---------- -------------------- --------- 
  training            normal,val_pq_restr+    normal 

1. Test that a QOS limit prevents over-allocation

Create a test QOS called tiny that allows a maximum of 2 CPUs per user total. Assign it to a test user. Submit two single-CPU jobs (they should start), then submit a third and confirm it is held pending with a QOS limit reason.

Hint / Solution

# Create the restrictive QOS
sacctmgr add qos tiny set MaxTRESPerUser=cpu=2 MaxJobsPerUser=10

# Assign to a test user
sacctmgr modify user testuser set qos=tiny defaultqos=tiny

# As testuser, submit jobs:
su - testuser
sbatch --qos=tiny --wrap="sleep 300" -n 1
sbatch --qos=tiny --wrap="sleep 300" -n 1
sbatch --qos=tiny --wrap="sleep 300" -n 1

# Check status -- the third job should be pending
squeue -u testuser -o "%.8i %.2t %.20R"
# Expected reason: (QOSMaxCpuPerUserLimit) or (QOSMaxTRESPerUser)

# Clean up
scancel -u testuser
sacctmgr delete qos tiny

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ squeue -u testuser1 -o '%.8i %.2t %.30R'
   JOBID ST               NODELIST(REASON)
     104 CF             batch-dy-compute-2
     105 PD        (QOSMaxCpuPerUserLimit)
     103  R             batch-dy-compute-1

1. Classify a cluster's scheduling policy from read-outs

Using only read-only commands, decide which of these a cluster is: strict FIFO (minimal policy), fair-share, or QOS-priority with an honour-system queue jump. Name the evidence for each line of your verdict.

Hint / Solution

# QOS rows: do they differ only by Priority?
sacctmgr show qos format=name,priority,maxwall,grptres

# Fair-share vs QOS weight: is the QOS term dominant?
scontrol show config | grep -E 'PriorityType|PriorityWeight'
sshare -a -l

# Who may target which QOS and partition (-a shows hidden partitions)
scontrol -a show partition | grep -E 'PartitionName|Hidden|AllowGroups|AllowQos|MaxTime'
sacctmgr show assoc format=user,account,qos,defaultqos

# Minimal-policy signature
scontrol show config | grep -E 'SchedulerType|PriorityType|PreemptMode|SelectTypeParameters'

Read it as: - `sched/builtin` + `priority/basic` + `PreemptMode=OFF` + `MaxTime=UNLIMITED` + `CR_CPU` = strict FIFO; there is no fair-share to inspect. - `priority/multifactor` with `PriorityWeightQOS` orders of magnitude above `PriorityWeightFairShare`, QOS rows that differ only in `Priority`, and `AllowQos=ALL` = any user can queue-jump with `--qos=`; `sshare -a -l` shows FairShare varying while pending jobs still order by QOS. - Fair-share: weights of comparable magnitude, QOS carrying limits (`MaxTRESPerUser`, `MaxWall`) instead of only `Priority`. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sacctmgr show qos format=name,priority,maxwall,grptres
      Name   Priority     MaxWall       GrpTRES 
---------- ---------- ----------- ------------- 
    normal          0                           
      high        100                           
       low         10                           

$ scontrol show config | grep -E 'PriorityType|PriorityWeight'
PriorityType            = priority/multifactor
PriorityWeightAge       = 1000
PriorityWeightAssoc     = 0
PriorityWeightFairShare = 10000
PriorityWeightJobSize   = 0
PriorityWeightPartition = 1000
PriorityWeightQOS       = 10000
PriorityWeightTRES      = (null)

$ sshare -a -l 2>&1 | head -20
Account                    User  RawShares  NormShares    RawUsage   NormUsage  EffectvUsage  FairShare    LevelFS                    GrpTRESMins                    TRESRunMins 
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------ 
root                                          0.000000         834                  1.000000                                                      cpu=0,mem=0,energy=0,node=0,b+ 
 root                      root          1    0.006579           0    0.000000      0.000000   1.000000        inf                                cpu=0,mem=0,energy=0,node=0,b+ 
 pcdefault                               1    0.006579         799    0.958109      0.958109              0.006867                                cpu=0,mem=0,energy=0,node=0,b+ 
  pcdefault            ec2-user          1    0.500000         799    0.958109      1.000000   0.250000   0.500000                                cpu=0,mem=0,energy=0,node=0,b+ 
  pcdefault               slurm          1    0.500000           0    0.000000      0.000000   0.500000        inf                                cpu=0,mem=0,energy=0,node=0,b+ 
 research                               50    0.328947          34    0.041891      0.041891              7.852417                                cpu=0,mem=0,energy=0,node=0,b+ 
  research            testuser1          1    1.000000          34    0.041891      1.000000   0.750000   1.000000                                cpu=0,mem=0,energy=0,node=0,b+ 
 training                              100    0.657895           0    0.000000      0.000000                   inf                                cpu=0,mem=0,energy=0,node=0,b+ 

$ scontrol -a show partition | grep -E 'PartitionName|Hidden|AllowGroups|AllowQos|MaxTime'
PartitionName=batch
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
PartitionName=debug
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
PartitionName=gpu
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
PartitionName=highmem
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED

$ sacctmgr show assoc format=user,account,qos,defaultqos
      User    Account                  QOS   Def QOS 
---------- ---------- -------------------- --------- 
                 root               normal           
      root       root               normal           
            pcdefault               normal           
  ec2-user  pcdefault               normal           
     slurm  pcdefault               normal           
             research               normal           
 testuser1   research               normal    normal 
             training               normal    normal 

$ scontrol show config | grep -E 'SchedulerType|PriorityType|PreemptMode|SelectTypeParameters'
PreemptMode             = OFF
PriorityType            = priority/multifactor
SchedulerType           = sched/backfill
SelectTypeParameters    = CR_CPU_MEMORY

References

• SchedMD: slurm.conf -- Partition Configuration
• SchedMD: Quality of Service (QOS)
• SchedMD: Resource Limits
• SchedMD: Preemption
• SchedMD: sacctmgr man page