Accounts & Fairshare

<!-- audience: L4, L5 --> <!-- deployment: all --> <!-- sge-source: SGE Admin 06-Policies (share tree, functional) --> # Accounts & Fairshare ## Account Hierarchy Slurm organizes resource tracking in a tree: ``` Cluster └── Account (organization, department, or lab) └── User (individual researcher) └── Association (user + account + optional partition) ``` An **association** ties a user to an account (and optionally a partition). Each association can have its own fairshare value and resource limits. --- ## Managing Accounts with sacctmgr > **ParallelCluster Note:** All `sacctmgr` commands work identically -- run them on the head node after cluster creation. Accounting is enabled by default with a local MariaDB instance. > **PCS Note:** The `sacctmgr` commands below work the same way, but run them from a login node. Accounting must be enabled at cluster creation time via the PCS API; it cannot be added after the fact. ### Create Accounts ```bash # Top-level organizational accounts sacctmgr add account biology Description="Biology Department" Organization=biology sacctmgr add account chemistry Description="Chemistry Department" Organization=chemistry # Sub-accounts (labs within departments) sacctmgr add account smith_lab parent=biology Description="Smith Lab" sacctmgr add account jones_lab parent=biology Description="Jones Lab" sacctmgr add account chem_comp parent=chemistry Description="Computational Chemistry" ``` ### Add Users ```bash # Add user to an account sacctmgr add user jdoe account=smith_lab # User in multiple accounts (one must be default) sacctmgr add user jdoe account=chem_comp sacctmgr modify user jdoe set defaultaccount=smith_lab # Add with specific QOS access sacctmgr add user jdoe account=smith_lab qos=normal,high defaultqos=normal ``` ### View Associations ```bash # Tree view of all associations sacctmgr show association format=Account,User,Fairshare,MaxJobs,MaxTRES tree withd # Specific user sacctmgr show association where user=jdoe format=Cluster,Account,User,Fairshare,QOS ``` ### Set Resource Limits on Accounts ```bash # Limit the biology department to 2048 CPUs total sacctmgr modify account biology set GrpTRES=cpu=2048 # Limit Smith Lab to 512 CPUs and 8 GPUs sacctmgr modify account smith_lab set GrpTRES=cpu=512,gres/gpu=8 # Per-user limits within an account sacctmgr modify user jdoe account=smith_lab set MaxTRESPerUser=cpu=64,gres/gpu=2 MaxJobs=20 ``` --- ## Fairshare Fairshare ensures that over time, each account gets its designated share of cluster resources. Users who have consumed less than their share get higher priority; those who've consumed more get lower priority. ### How It Works 1. Each account/user is assigned a number of **shares** (relative weight) 2. Slurm tracks actual **usage** (CPU-hours, GPU-hours, etc.) 3. The fairshare factor is computed by comparing shares to usage 4. Jobs from under-served accounts get higher priority ### Fair Tree (Default Since 19.05) The default algorithm. Key property: **if account A has a better fairshare factor than sibling account B, all users under A have higher fairshare than all users under B.** This prevents coordinators from accidentally harming their users relative to other accounts. ```ini # slurm.conf -- Fair Tree is the default, but to be explicit: PriorityType=priority/multifactor # PriorityFlags=NO_FAIR_TREE # Only if you want the old "Classic" algorithm ``` ### Configuring Shares ```bash # Set shares at the account level sacctmgr modify account biology set fairshare=60 sacctmgr modify account chemistry set fairshare=40 # Sub-accounts sacctmgr modify account smith_lab set fairshare=30 sacctmgr modify account jones_lab set fairshare=30 # User-level shares (within their account) sacctmgr modify user jdoe account=smith_lab set fairshare=10 sacctmgr modify user asmith account=smith_lab set fairshare=10 ``` Shares are **relative within a level**. Biology has 60/(60+40) = 60% of the cluster target. Smith Lab has 30/(30+30) = 50% of biology's share = 30% of cluster. ### Viewing Fairshare ```bash # Current fairshare state sshare -l Account User RawShares NormShares RawUsage EffectvUsage FairShare LevelFS ---------------- ---------- ---------- ---------- ----------- ------------ --------- -------- root biology 60 0.600000 123456 0.450000 0.650000 1.333333 smith_lab 30 0.500000 45678 0.370000 0.700000 1.351351 smith_lab jdoe 10 0.500000 23456 0.513699 0.520000 0.973236 smith_lab asmith 10 0.500000 22222 0.486301 0.540000 1.028169 jones_lab 30 0.500000 77778 0.630000 0.420000 0.793651 chemistry 40 0.400000 150000 0.550000 0.450000 0.727273 ``` Key columns: - **NormShares:** Normalized share (0-1) within this level - **EffectvUsage:** Normalized actual usage - **FairShare:** Final fairshare factor (higher = higher priority) - **LevelFS:** Level fairshare (NormShares/EffectvUsage) -- used by Fair Tree ### Fairshare Decay Usage decays over time so past heavy usage doesn't penalize users forever: ```ini # slurm.conf PriorityDecayHalfLife=14-0 # Usage halves every 14 days PriorityUsageResetPeriod=NONE # Or MONTHLY, QUARTERLY, etc. ``` ### Reset Usage ```bash # Reset a user's usage (emergency) sacctmgr modify user jdoe set rawusage=0 # Reset an entire account sacctmgr modify account smith_lab set rawusage=0 ``` --- ## Priority Weights Fairshare is one component of the multifactor priority. Configure how much each factor matters: ```ini # slurm.conf PriorityWeightFairshare=100000 # Dominant factor PriorityWeightAge=1000 # Waiting time bonus PriorityWeightPartition=10000 # Partition-based priority PriorityWeightJobSize=1000 # Larger jobs get slight boost PriorityWeightQOS=10000 # QOS-based priority ``` View priority breakdown: ```bash $ sprio -l JOBID PARTITION USER PRIORITY AGE FAIRSHARE JOBSIZE PARTITION QOS 12345 batch jdoe 50123 500 45000 1000 3000 623 12346 batch asmith 48500 200 43000 1000 3000 1300 ``` --- ## Example: Setting Up a New Lab ```bash # 1. Create the account sacctmgr add account new_lab parent=biology \ Description="New Lab - Dr. Garcia" fairshare=20 # 2. Add users sacctmgr add user garcia account=new_lab \ fairshare=10 qos=normal,high defaultqos=normal adminlevel=operator sacctmgr add user postdoc1 account=new_lab fairshare=5 sacctmgr add user student1 account=new_lab fairshare=3 # 3. Set limits sacctmgr modify account new_lab set \ GrpTRES=cpu=256,gres/gpu=4 \ MaxJobs=100 # 4. Verify sacctmgr show association where account=new_lab format=Account,User,Fairshare,QOS,GrpTRES tree ```

Exercises

1. Create a three-level account hierarchy

Create an organizational structure: a top-level account research, a department account structural_biology under it, and a lab account cryo_lab under the department. Add two users (pi_chen and postdoc_kim) to cryo_lab with different fairshare weights.

Hint / Solution

# Create the hierarchy
sacctmgr add account research Description="Research Division"
sacctmgr add account structural_biology parent=research Description="Structural Biology Dept"
sacctmgr add account cryo_lab parent=structural_biology Description="Cryo-EM Lab"

# Add users with different share weights
sacctmgr add user pi_chen account=cryo_lab fairshare=20 qos=normal defaultqos=normal
sacctmgr add user postdoc_kim account=cryo_lab fairshare=10 qos=normal defaultqos=normal

# Verify the tree
sacctmgr show association where account=cryo_lab format=Account,User,Fairshare tree withd

1. Inspect fairshare values with sshare

Use sshare to view the current fairshare state for an account hierarchy. Identify which account has the best LevelFS (most under-served relative to its share) and which has the worst.

Hint / Solution

# View full fairshare tree
sshare -l -A research

# Key columns to examine:
# - NormShares: the account's share as a fraction of its siblings
# - EffectvUsage: normalized actual usage
# - LevelFS: NormShares / EffectvUsage (higher = more under-served)

# To sort by fairshare factor (most under-served first):
sshare -l | sort -k7 -rn | head -20

1. Modify share allocation for a department

Change the fairshare allocation so that structural_biology gets 40 shares and a sibling account gets 60 shares. Verify the normalized shares reflect the new 40/60 split.

Hint / Solution

# Set shares
sacctmgr modify account structural_biology set fairshare=40
sacctmgr modify account chemistry set fairshare=60    # sibling account

# Verify normalized shares
sshare -l -A structural_biology,chemistry
# structural_biology NormShares should be ~0.400000
# chemistry NormShares should be ~0.600000

1. Generate a usage report with sreport

Use sreport to produce a report showing CPU-hour usage by account for the current month. Then generate a second report showing the top 10 users by CPU hours.

Hint / Solution

# Usage by account for the current month
sreport cluster AccountUtilizationByUser start=2026-04-01 --tres=cpu

# Top 10 users by CPU hours
sreport user TopUsage start=2026-04-01 --tres=cpu TopCount=10

# For GPU usage by account:
sreport cluster AccountUtilizationByUser start=2026-04-01 --tres=gres/gpu

References

• SchedMD: Accounting and Resource Limits
• SchedMD: Fair Tree Fairshare Algorithm
• SchedMD: Classic Fairshare Algorithm
• SchedMD: Multifactor Priority Plugin
• SchedMD: sacctmgr man page
• SchedMD: sshare man page
• SchedMD: sprio man page