Accounts & Fairshare

<!-- audience: L4, L5 --> <!-- deployment: all --> <!-- sge-source: SGE Admin 06-Policies (share tree, functional) --> # Accounts & Fairshare ## Account Hierarchy Slurm organizes resource tracking in a tree: ``` Cluster └── Account (organization, department, or lab) └── User (individual researcher) └── Association (user + account + optional partition) ``` An **association** ties a user to an account (and optionally a partition). Each association can have its own fairshare value and resource limits. --- ## Managing Accounts with sacctmgr > **ParallelCluster Note:** All `sacctmgr` commands work identically -- run them on the head node after cluster creation. Accounting is enabled by default with a local MariaDB instance. > **PCS Note:** The `sacctmgr` commands below work the same way, but run them from a login node. Accounting must be enabled at cluster creation time via the PCS API; it cannot be added after the fact. ### Create Accounts ```bash # Top-level organizational accounts sacctmgr add account biology Description="Biology Department" Organization=biology sacctmgr add account chemistry Description="Chemistry Department" Organization=chemistry # Sub-accounts (labs within departments) sacctmgr add account smith_lab parent=biology Description="Smith Lab" sacctmgr add account jones_lab parent=biology Description="Jones Lab" sacctmgr add account chem_comp parent=chemistry Description="Computational Chemistry" ``` ### Add Users ```bash # Add user to an account sacctmgr add user jdoe account=smith_lab # User in multiple accounts (one must be default) sacctmgr add user jdoe account=chem_comp sacctmgr modify user jdoe set defaultaccount=smith_lab # Add with specific QOS access sacctmgr add user jdoe account=smith_lab qos=normal,high defaultqos=normal ``` --- ## View Associations ```bash # Tree view of all associations sacctmgr show association format=Account,User,Fairshare,MaxJobs,MaxTRES tree withd # Specific user sacctmgr show association where user=jdoe format=Cluster,Account,User,Fairshare,QOS ``` ## Set Resource Limits on Accounts ```bash # Limit the biology department to 2048 CPUs total sacctmgr modify account biology set GrpTRES=cpu=2048 # Limit Smith Lab to 512 CPUs and 8 GPUs sacctmgr modify account smith_lab set GrpTRES=cpu=512,gres/gpu=8 # Per-user limits within an account sacctmgr modify user jdoe account=smith_lab set MaxTRESPerUser=cpu=64,gres/gpu=2 MaxJobs=20 ``` --- ## Fairshare Fairshare ensures that over time, each account gets its designated share of cluster resources. Users who have consumed less than their share get higher priority; those who've consumed more get lower priority. ### How It Works 1. Each account/user is assigned a number of **shares** (relative weight) 2. Slurm tracks actual **usage** (CPU-hours, GPU-hours, etc.) 3. The fairshare factor is computed by comparing shares to usage 4. Jobs from under-served accounts get higher priority ### Fair Tree (Default Since 19.05) The default algorithm. Key property: **if account A has a better fairshare factor than sibling account B, all users under A have higher fairshare than all users under B.** This prevents coordinators from accidentally harming their users relative to other accounts. ```ini # slurm.conf -- Fair Tree is the default, but to be explicit: PriorityType=priority/multifactor # PriorityFlags=NO_FAIR_TREE # Only if you want the old "Classic" algorithm ``` ### Configuring Shares ```bash # Set shares at the account level sacctmgr modify account biology set fairshare=60 sacctmgr modify account chemistry set fairshare=40 # Sub-accounts sacctmgr modify account smith_lab set fairshare=30 sacctmgr modify account jones_lab set fairshare=30 # User-level shares (within their account) sacctmgr modify user jdoe account=smith_lab set fairshare=10 sacctmgr modify user asmith account=smith_lab set fairshare=10 ``` Shares are **relative within a level**. Biology has 60/(60+40) = 60% of the cluster target. Smith Lab has 30/(30+30) = 50% of biology's share = 30% of cluster. --- ## Viewing Fairshare ```bash # Current fairshare state sshare -l Account User RawShares NormShares RawUsage EffectvUsage FairShare LevelFS ---------------- ---------- ---------- ---------- ----------- ------------ --------- -------- root biology 60 0.600000 123456 0.450000 0.650000 1.333333 smith_lab 30 0.500000 45678 0.370000 0.700000 1.351351 smith_lab jdoe 10 0.500000 23456 0.513699 0.520000 0.973236 smith_lab asmith 10 0.500000 22222 0.486301 0.540000 1.028169 jones_lab 30 0.500000 77778 0.630000 0.420000 0.793651 chemistry 40 0.400000 150000 0.550000 0.450000 0.727273 ``` Key columns: - **NormShares:** Normalized share (0-1) within this level - **EffectvUsage:** Normalized actual usage - **FairShare:** Final fairshare factor (higher = higher priority) - **LevelFS:** Level fairshare (NormShares/EffectvUsage) -- used by Fair Tree --- ## Fairshare Decay and Resets ### Fairshare Decay Usage decays over time so past heavy usage doesn't penalize users forever: ```ini # slurm.conf PriorityDecayHalfLife=14-0 # Usage halves every 14 days PriorityUsageResetPeriod=NONE # Or MONTHLY, QUARTERLY, etc. ``` ### Reset Usage ```bash # Reset a user's usage (emergency) sacctmgr modify user jdoe set rawusage=0 # Reset an entire account sacctmgr modify account smith_lab set rawusage=0 ``` --- ## Priority Weights Fairshare is one component of the multifactor priority. Configure how much each factor matters: ```ini # slurm.conf PriorityWeightFairshare=100000 # Dominant factor PriorityWeightAge=1000 # Waiting time bonus PriorityWeightPartition=10000 # Partition-based priority PriorityWeightJobSize=1000 # Larger jobs get slight boost PriorityWeightQOS=10000 # QOS-based priority ``` View priority breakdown: ```bash $ sprio -l JOBID PARTITION USER PRIORITY AGE FAIRSHARE JOBSIZE PARTITION QOS 12345 batch jdoe 50123 500 45000 1000 3000 623 12346 batch asmith 48500 200 43000 1000 3000 1300 ``` --- ## Example: Setting Up a New Lab ```bash # 1. Create the account sacctmgr add account new_lab parent=biology \ Description="New Lab - Dr. Garcia" fairshare=20 # 2. Add users sacctmgr add user garcia account=new_lab \ fairshare=10 qos=normal,high defaultqos=normal adminlevel=operator sacctmgr add user postdoc1 account=new_lab fairshare=5 sacctmgr add user student1 account=new_lab fairshare=3 # 3. Set limits sacctmgr modify account new_lab set \ GrpTRES=cpu=256,gres/gpu=4 \ MaxJobs=100 # 4. Verify sacctmgr show association where account=new_lab format=Account,User,Fairshare,QOS,GrpTRES tree ```

Exercises

1. Create a three-level account hierarchy

Create an organizational structure: a top-level account research, a department account structural_biology under it, and a lab account cryo_lab under the department. Add two users (pi_chen and postdoc_kim) to cryo_lab with different fairshare weights.

Hint / Solution

# Create the hierarchy
sacctmgr add account research Description="Research Division"
sacctmgr add account structural_biology parent=research Description="Structural Biology Dept"
sacctmgr add account cryo_lab parent=structural_biology Description="Cryo-EM Lab"

# Add users with different share weights
sacctmgr add user pi_chen account=cryo_lab fairshare=20 qos=normal defaultqos=normal
sacctmgr add user postdoc_kim account=cryo_lab fairshare=10 qos=normal defaultqos=normal

# Verify the tree
sacctmgr show association where account=cryo_lab format=Account,User,Fairshare tree withd

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sudo /opt/slurm/bin/sacctmgr -i add account val_research Description='Research Division'
 Adding Account(s)
  val_research
 Settings
  Description     = research division
  Organization    = Parent/Account Name
 Associations =
  C = slurm-training-val A = val_research        
 Settings

$ sudo /opt/slurm/bin/sacctmgr -i add account val_structural_biology parent=val_research Description='Structural Biology Dept'
 Adding Account(s)
  val_structural_biology
 Settings
  Description     = structural biology dept
  Organization    = Parent/Account Name
 Associations =
  C = slurm-training-val A = val_structural_biology
 Settings
  Parent        = val_research

$ sudo /opt/slurm/bin/sacctmgr -i add account val_cryo_lab parent=val_structural_biology Description='Cryo-EM Lab'
 Adding Account(s)
  val_cryo_lab
 Settings
  Description     = cryo-em lab
  Organization    = Parent/Account Name
 Associations =
  C = slurm-training-val A = val_cryo_lab        
 Settings
  Parent        = val_structural_biology

$ sudo /opt/slurm/bin/sacctmgr -i add user val_pi_chen account=val_cryo_lab fairshare=20 qos=normal defaultqos=normal
 Adding User(s)
  val_pi_chen
 Settings
 Associations =
  C = slurm-training-val A = val_cryo_lab         U = val_pi_chen
 Non Default Settings
  Fairshare     = 20
  QOS           = normal
  DefQOS        = normal

$ sudo /opt/slurm/bin/sacctmgr -i add user val_postdoc_kim account=val_cryo_lab fairshare=10 qos=normal defaultqos=normal
 Adding User(s)
  val_postdoc_kim
 Settings
 Associations =
  C = slurm-training-val A = val_cryo_lab         U = val_postdoc_kim
 Non Default Settings
  Fairshare     = 10
  QOS           = normal
  DefQOS        = normal

$ sacctmgr show association where account=val_cryo_lab format=Account,User,Fairshare tree withd
Account                    User     Share 
-------------------- ---------- --------- 
val_cryo_lab                            1 
 val_cryo_lab        val_pi_ch+        20 
 val_cryo_lab        val_postd+        10 

$ sacctmgr show user val_pi_chen,val_postdoc_kim format=User%-20,DefaultAccount%-20
User                 Def Acct             
-------------------- -------------------- 
val_pi_chen          val_cryo_lab         
val_postdoc_kim      val_cryo_lab         

1. Inspect fairshare values with sshare

Use sshare to view the current fairshare state for an account hierarchy. Identify which account has the best LevelFS (most under-served relative to its share) and which has the worst.

Hint / Solution

# View full fairshare tree
sshare -l -A research

# Key columns to examine:
# - NormShares: the account's share as a fraction of its siblings
# - EffectvUsage: normalized actual usage
# - LevelFS: NormShares / EffectvUsage (higher = more under-served)

# To sort by fairshare factor (most under-served first):
sshare -l | sort -k7 -rn | head -20

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sshare -l | head -20
Account                    User  RawShares  NormShares    RawUsage   NormUsage  EffectvUsage  FairShare    LevelFS                    GrpTRESMins                    TRESRunMins 
-------------------- ---------- ---------- ----------- ----------- ----------- ------------- ---------- ---------- ------------------------------ ------------------------------ 
root                                          0.000000        7172                  1.000000                                                      cpu=0,mem=0,energy=0,node=0,b+ 
 pcdefault                               1    0.006579        6891    0.960861      0.960861              0.006847                                cpu=0,mem=0,energy=0,node=0,b+ 
  pcdefault            ec2-user          1    0.500000        6891    0.960861      1.000000   0.250000   0.500000                                cpu=0,mem=0,energy=0,node=0,b+ 
 research                               50    0.328947         280    0.039139      0.039139              8.404634                                cpu=0,mem=0,energy=0,node=0,b+ 
 training                              100    0.657895           0    0.000000      0.000000                   inf                                cpu=0,mem=0,energy=0,node=0,b+ 

1. Modify share allocation for a department

Change the fairshare allocation so that structural_biology gets 40 shares and a sibling account gets 60 shares. Verify the normalized shares reflect the new 40/60 split.

Hint / Solution

# Set shares
sacctmgr modify account structural_biology set fairshare=40
sacctmgr modify account chemistry set fairshare=60    # sibling account

# Verify normalized shares
sshare -l -A structural_biology,chemistry
# structural_biology NormShares should be ~0.400000
# chemistry NormShares should be ~0.600000

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sudo /opt/slurm/bin/sacctmgr -i modify account val_structural_biology set fairshare=40
 Modified account associations...
  C = slurm-training-val A = val_structural_biology of val_research

$ sudo /opt/slurm/bin/sacctmgr -i modify account val_chemistry set fairshare=60
 Modified account associations...
  C = slurm-training-val A = val_chemistry of val_research

$ sacctmgr show account val_structural_biology,val_chemistry format=Account%-25,Description%-25,Organization%-25
Account                   Descr                     Org                       
------------------------- ------------------------- ------------------------- 
val_chemistry             val_chemistry             val_research              
val_structural_biology    val_structural_biology    val_research              

1. Generate a usage report with sreport

Use sreport to produce a report showing CPU-hour usage by account for the current month. Then generate a second report showing the top 10 users by CPU hours.

Hint / Solution

# Usage by account for the current month
sreport cluster AccountUtilizationByUser start=2026-04-01 --tres=cpu

# Top 10 users by CPU hours
sreport user TopUsage start=2026-04-01 --tres=cpu TopCount=10

# For GPU usage by account:
sreport cluster AccountUtilizationByUser start=2026-04-01 --tres=gres/gpu

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sreport cluster AccountUtilizationByUser start=now-7days --tres=cpu | head -15
--------------------------------------------------------------------------------
Cluster/Account/User Utilization 2026-04-08T02:00:00 - 2026-04-15T00:59:59 (601200 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
  Cluster         Account     Login     Proper Name      TRES Name     Used 
--------- --------------- --------- --------------- -------------- -------- 
slurm-tr+            root                                      cpu      286 
slurm-tr+       pcdefault                                      cpu      281 
slurm-tr+       pcdefault  ec2-user AWS ParallelCl+            cpu      281 
slurm-tr+        research                                      cpu        5 
slurm-tr+        research testuser1                            cpu        5 

$ sreport user TopUsage start=now-7days --tres=cpu TopCount=10 | head -15
--------------------------------------------------------------------------------
Top 10 Users 2026-04-08T02:00:00 - 2026-04-15T00:59:59 (601200 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
  Cluster     Login     Proper Name         Account      TRES Name     Used 
--------- --------- --------------- --------------- -------------- -------- 
slurm-tr+  ec2-user AWS ParallelCl+       pcdefault            cpu      281 
slurm-tr+ testuser1                        research            cpu        5 

References

• SchedMD: Accounting and Resource Limits
• SchedMD: Fair Tree Fairshare Algorithm
• SchedMD: Classic Fairshare Algorithm
• SchedMD: Multifactor Priority Plugin
• SchedMD: sacctmgr man page
• SchedMD: sshare man page
• SchedMD: sprio man page