Policies & Priority

<!-- audience: L4, L5 --> <!-- deployment: all --> <!-- sge-source: SGE Admin 06-Policies (reservation, backfilling, urgency) --> # Policies & Priority ## Multifactor Priority Slurm's default priority plugin computes a composite score from multiple factors: ``` Job Priority = PriorityWeightAge * AgeFactor + PriorityWeightFairshare * FairshareFactor + PriorityWeightJobSize * JobSizeFactor + PriorityWeightPartition * PartitionFactor + PriorityWeightQOS * QOSFactor + PriorityWeightAssoc * AssocFactor + PriorityWeightTRES * TRESFactors - nice ``` ### Configuration ```ini # slurm.conf PriorityType=priority/multifactor PriorityWeightFairshare=100000 # Dominant factor PriorityWeightAge=1000 # Time in queue PriorityWeightPartition=10000 # Partition importance PriorityWeightJobSize=1000 # Favor larger jobs (helps utilization) PriorityWeightQOS=10000 # QOS priority PriorityDecayHalfLife=14-0 # Usage decays over 14 days PriorityMaxAge=7-0 # Age factor maxes out after 7 days ``` > **ParallelCluster Note:** Priority-related slurm.conf parameters (`PriorityType`, `PriorityWeightAge`, etc.) must be set via `CustomSlurmSettings` in your cluster YAML config. Direct slurm.conf edits are overwritten on cluster updates. QOS configuration via `sacctmgr` works normally. > **PCS Note:** Partition priority is set through the PCS API or console at the queue level. Slurm-level priority tuning (multifactor weights, QOS) can be applied via custom Slurm settings in your PCS cluster configuration. --- ## Viewing Priority ```bash # See priority breakdown for pending jobs $ sprio JOBID PARTITION USER PRIORITY AGE FAIRSHARE JOBSIZE PARTITION QOS 12345 batch jdoe 85123 500 75000 1000 8000 623 12346 batch asmith 82500 200 73000 1000 8000 300 # Detailed view $ sprio -l -j 12345 # See configured weights $ sprio -w ``` --- ## Scheduling Order The scheduler evaluates jobs in this order: 1. Jobs that can **preempt** (if preemption is enabled) 2. Jobs with an **advanced reservation** 3. **Partition PriorityTier** (higher tiers evaluated first) 4. **Job priority** (the multifactor score) 5. **Job submit time** (FIFO tiebreaker) 6. **Job ID** (final tiebreaker) This means a debug partition with `PriorityTier=200` is evaluated before batch with `PriorityTier=100`, regardless of individual job priorities. --- ## Preemption Preemption allows high-priority jobs to displace running lower-priority jobs. ### Preemption Modes | Mode | Action | |------|--------| | `CANCEL` | Kill the preempted job | | `REQUEUE` | Requeue the preempted job (restarts from scratch) | | `SUSPEND` | Suspend the preempted job (resume later) | | `GANG` | Time-slice between preempting and preempted jobs | ### Configuration ```ini # slurm.conf PreemptType=preempt/qos # QOS-based preemption PreemptMode=REQUEUE # Default action PreemptExemptTime=00:05:00 # Jobs running <5 min are exempt ``` --- ## Preemption Methods ### QOS-Based Preemption ```bash # high QOS can preempt normal and low sacctmgr modify qos high set Preempt=normal,low PreemptMode=REQUEUE GraceTime=120 # urgent QOS can preempt everything sacctmgr modify qos urgent set Preempt=normal,low,high PreemptMode=CANCEL ``` ### Partition-Based Preemption ```ini # slurm.conf PreemptType=preempt/partition_prio PartitionName=premium Nodes=node[001-100] PriorityTier=200 PreemptMode=REQUEUE PartitionName=standard Nodes=node[001-100] PriorityTier=100 PartitionName=scavenger Nodes=node[001-100] PriorityTier=50 ``` Jobs in `premium` preempt `standard` and `scavenger`. Jobs in `standard` preempt `scavenger`. --- ## Backfill Scheduling The backfill scheduler is critical for cluster utilization. It allows lower-priority jobs to start on reserved resources if they'll finish before the higher-priority job needs them. ```ini # slurm.conf SchedulerType=sched/backfill SchedulerParameters=bf_max_job_test=5000,bf_interval=30,bf_resolution=600 ``` | Parameter | Description | Default | |-----------|-------------|---------| | `bf_interval` | Seconds between backfill cycles | 30 | | `bf_max_job_test` | Max jobs evaluated per cycle | 500 | | `bf_resolution` | Time granularity for backfill (seconds) | 60 | **Why walltime matters for users:** The backfill scheduler uses job time limits to predict resource availability. Jobs with accurate (shorter) time limits are more likely to be backfilled. This is why user training should emphasize setting reasonable `--time` values. --- ## Reservations Reserve specific resources for maintenance, special events, or guaranteed access. ### Create a Reservation ```bash # Maintenance window scontrol create reservation ReservationName=maint_apr15 \ StartTime=2026-04-15T02:00:00 Duration=120 \ Nodes=node[001-010] User=root Flags=MAINT,IGNORE_JOBS # Reserved for a specific user/account scontrol create reservation ReservationName=workshop \ StartTime=2026-04-20T09:00:00 EndTime=2026-04-20T17:00:00 \ NodeCnt=10 Accounts=training_account # TRES-based reservation (e.g., reserve 200 GPUs) scontrol create reservation ReservationName=gpu_block \ StartTime=now Duration=480 \ TRES=gres/gpu=200 User=jdoe ``` --- ## Manage Reservations ```bash # List reservations scontrol show reservation # Users submit to a reservation sbatch --reservation=workshop my_job.sh # Modify scontrol update ReservationName=maint_apr15 Duration=180 # Delete scontrol delete ReservationName=workshop ``` --- ## Reservation Flags | Flag | Purpose | |------|---------| | `MAINT` | Maintenance reservation (no user jobs) | | `IGNORE_JOBS` | Allow reservation to overlap running jobs | | `FLEX` | Allow job to use more or fewer nodes than reserved | | `DAILY` / `WEEKLY` | Recurring reservation | --- ## Resource Quotas via QOS Beyond the QOS limits covered in [Partitions & QOS](partitions-qos.md), common quota patterns: ### Limit Total Cluster Usage Per Account ```bash sacctmgr modify account labA set GrpTRES=cpu=1024,gres/gpu=16 ``` ### Limit Per-User Concurrent Usage ```bash sacctmgr modify qos normal set MaxTRESPerUser=cpu=256,gres/gpu=4 MaxJobsPerUser=50 ``` ### Limit Job Size ```bash sacctmgr modify qos normal set MaxTRESPerJob=cpu=64,node=4 ``` ### Implement "Burst" Access A "burst" QOS that allows short periods of heavy usage: ```bash sacctmgr add qos burst set \ Priority=500 \ MaxTRESPerUser=cpu=512 \ MaxWallDurationPerJob=04:00:00 \ UsageFactor=2.0 # Charges double for fairshare ``` --- ## Policy Design Recommendations 1. **Start simple.** Fairshare with equal shares for all accounts is a good starting point. Add complexity only when needed. 2. **Fairshare is usually sufficient.** Don't add preemption unless users explicitly need it. Preemption adds complexity and can cause work loss. 3. **Use partition PriorityTier for evaluation order** (debug first, then interactive, then batch). Use QOS for priority within a tier. 4. **Monitor with `sprio` and `sshare`.** If users don't understand why their jobs are waiting, the policy is too complex. 5. **Document your policies.** Publish a clear page explaining how scheduling works, what the partitions/QOS mean, and how to check priority.

Exercises

1. View priority breakdown for pending jobs

Use sprio to examine the multifactor priority components for all pending jobs. Identify which factor (Age, Fairshare, JobSize, Partition, QOS) is contributing the most to each job's priority score.

Hint / Solution

# Show priority breakdown for all pending jobs
sprio -l

# Show normalized (0.0-1.0) priority factors for easier comparison
sprio -n -l

# Show priority for a specific user's pending jobs
sprio -u jsmith -l

# Compare with sshare to see fairshare standings
sshare -a -l

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sprio -l
          JOBID PARTITION     USER  ACCOUNT   PRIORITY       SITE        AGE      ASSOC  FAIRSHARE    JOBSIZE  PARTITION    QOSNAME        QOS        NICE                 TRES

$ sprio -n -l
          JOBID PARTITION     USER  ACCOUNT PRIORITY   AGE        ASSOC      FAIRSHARE  JOBSIZE    PARTITION  QOSNAME    QOS        TRES                

$ sshare -a -l | head -10
Account                    User  RawShares  NormShares    RawUsage   NormUsage  EffectvUsage  FairShare    LevelFS                    GrpTRESMins                    TRESRunMins 
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------ 
root                                          0.000000        7187                  1.000000                                                      cpu=0,mem=0,energy=0,node=0,b+ 
 root                      root          1    0.006579           0    0.000000      0.000000   1.000000        inf                                cpu=0,mem=0,energy=0,node=0,b+ 
 pcdefault                               1    0.006579        6889    0.958455      0.958455              0.006864                                cpu=0,mem=0,energy=0,node=0,b+ 
  pcdefault            ec2-user          1    0.500000        6889    0.958455      1.000000   0.250000   0.500000                                cpu=0,mem=0,energy=0,node=0,b+ 
  pcdefault               slurm          1    0.500000           0    0.000000      0.000000   0.500000        inf                                cpu=0,mem=0,energy=0,node=0,b+ 
 research                               50    0.328947         298    0.041545      0.041545              7.917862                                cpu=0,mem=0,energy=0,node=0,b+ 
  research            testuser1          1    1.000000         298    0.041545      1.000000   0.750000   1.000000                                cpu=0,mem=0,energy=0,node=0,b+ 
 training                              100    0.657895           0    0.000000      0.000000                   inf                                cpu=0,mem=0,energy=0,node=0,b+ 

1. Create a test reservation

Create a short maintenance reservation on a single node for 30 minutes starting 1 hour from now. Verify the reservation exists, then delete it when done.

Hint / Solution

# Create a 30-minute reservation starting 1 hour from now
scontrol create reservation ReservationName=maint_test \
    StartTime=now+3600 Duration=00:30:00 \
    Nodes=cpu001 Users=root Flags=MAINT,IGNORE_JOBS

# Verify the reservation
scontrol show reservation maint_test

# List all reservations
scontrol show reservations

# Clean up — delete the test reservation
scontrol delete ReservationName=maint_test

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sudo /opt/slurm/bin/scontrol create reservation ReservationName=val_pp_resv        StartTime=now+3600 Duration=00:30:00        Nodes=debug-dy-small-1 Users=ec2-user Flags=MAINT,IGNORE_JOBS
Reservation created: val_pp_resv

$ scontrol show reservation val_pp_resv
ReservationName=val_pp_resv StartTime=2026-04-15T03:01:06 EndTime=2026-04-15T03:31:06 Duration=00:30:00
   Nodes=debug-dy-small-1 NodeCnt=1 CoreCnt=2 Features=(null) PartitionName=(null) Flags=MAINT,IGNORE_JOBS,SPEC_NODES
   TRES=cpu=2
   AllowedPartitions=(null) QOS=(null)
   Users=ec2-user Groups=(null) Accounts=(null) Licenses=(null) State=INACTIVE BurstBuffer=(null)
   MaxStartDelay=(null)

$ sudo /opt/slurm/bin/scontrol delete ReservationName=val_pp_resv

1. Check backfill scheduler status

Use sdiag to examine the scheduler's backfill statistics. Determine how many backfill cycles have run, the average cycle time, and whether any backfill cycles are taking too long (which can indicate configuration issues).

Hint / Solution

# Show full scheduler diagnostics
sdiag

# Key sections to look for:
#   Main schedule statistics — how often the main scheduler runs
#   Backfilling stats — cycle count, last cycle time, max cycle time
#
# Warning signs:
#   - Last cycle (usec) > 1,000,000 (1 second) — backfill is slow
#   - Backfilled jobs since start = 0 — backfill may be disabled
#   - Total backfilled jobs (since last stats cycle reset) growing slowly

# Reset statistics (useful after config changes)
sdiag -r

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sdiag | head -40
*******************************************************
sdiag output at Wed Apr 15 02:01:06 2026 (1776218466)
Data since      Wed Apr 15 02:01:02 2026 (1776218462)
*******************************************************
Server thread count:  1
RPC queue enabled:    0
Agent queue size:     0
Agent count:          0
Agent thread count:   0
DBD Agent queue size: 0

Jobs submitted: 1
Jobs started:   0
Jobs completed: 0
Jobs canceled:  1
Jobs failed:    0

Job states ts:  Wed Apr 15 02:01:02 2026 (1776218462)
Jobs pending:   0
Jobs running:   0

Main schedule statistics (microseconds):
 Last cycle:   9
 Max cycle:    16
 Total cycles: 2
 Mean cycle:   12
 Mean depth cycle:  0
 Last queue length: 0

Main scheduler exit:
 End of job queue: 2
 Hit default_queue_depth: 0
 Hit sched_max_job_start: 0
 Blocked on licenses: 0
 Hit max_rpc_cnt: 0
 Timeout (max_sched_time): 0

Backfilling stats
 Total backfilled jobs (since last slurm start): 0
 Total backfilled jobs (since last stats cycle start): 0

References

• SchedMD: Multifactor Priority Plugin
• SchedMD: Preemption
• SchedMD: Scheduling Configuration Guide
• SchedMD: Advanced Resource Reservation Guide
• SchedMD: sprio man page