Configuration

<!-- audience: L4 --> <!-- deployment: all --> <!-- sge-source: SGE Admin 03-Config --> # Configuration ## Configuration Files Overview | File | Purpose | Location | |------|---------|----------| | `slurm.conf` | Main configuration (nodes, partitions, scheduler, plugins) | `/etc/slurm/slurm.conf` | | `slurmdbd.conf` | Accounting daemon config (database connection) | `/etc/slurm/slurmdbd.conf` | | `cgroup.conf` | Control group settings (resource enforcement) | `/etc/slurm/cgroup.conf` | | `gres.conf` | Generic resource definitions (GPUs per node) | `/etc/slurm/gres.conf` | | `topology.conf` | Network topology for optimized scheduling | `/etc/slurm/topology.conf` | All config files must be consistent across nodes (use a shared filesystem or configless mode). --- ## slurm.conf Key Parameters ### Cluster Identity ```ini ClusterName=mycluster SlurmctldHost=mgmt01(10.0.0.1) SlurmctldHost=mgmt02(10.0.0.2) # Backup for HA ``` ### Resource Selection ```ini SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory ``` `select/cons_tres` (consumable TRES) is the recommended and default plugin. It tracks individual CPUs, memory, and GRES as consumable resources, enabling node sharing. --- ## slurm.conf Key Parameters (cont.) ### Scheduler ```ini SchedulerType=sched/backfill SchedulerParameters=bf_max_job_test=5000,bf_interval=30,bf_resolution=600 ``` The backfill scheduler is essential for utilization. It allows smaller/shorter jobs to start on resources reserved for larger pending jobs, as long as they don't delay the larger job's start time. ### Logging ```ini SlurmctldDebug=info # Options: quiet, fatal, error, info, verbose, debug, debug2-5 SlurmdDebug=info SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log ``` Increase to `debug` temporarily for troubleshooting. Don't run `debug` in production long-term -- it generates massive logs. ### Timeouts ```ini SlurmctldTimeout=120 # Seconds before backup takes over SlurmdTimeout=300 # Seconds before node marked DOWN MessageTimeout=10 # RPC timeout UnkillableStepTimeout=60 ``` --- ## Node Definitions ```ini # Default values inherited by all nodes NodeName=DEFAULT CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=256000 State=UNKNOWN # Standard compute nodes NodeName=node[001-048] NodeAddr=node[001-048] # High-memory nodes NodeName=bigmem[1-4] CPUs=128 RealMemory=1024000 # GPU nodes NodeName=gpu[01-08] CPUs=64 RealMemory=512000 Gres=gpu:a100:4 ``` Key node parameters: | Parameter | Description | |-----------|-------------| | `CPUs` | Total logical CPUs (cores x threads if hyperthreading) | | `RealMemory` | Physical RAM in MB (leave ~5% for OS) | | `Gres` | Generic resources (GPUs, etc.) | | `State` | Initial state (`UNKNOWN` lets slurmd report actual state) | | `Features` | Admin-defined tags for constraint matching | | `Weight` | Scheduling priority (lower = preferred) | ### Node Features Tag nodes for user `--constraint` matching: ```ini NodeName=node[001-024] Features=haswell,ethernet NodeName=node[025-048] Features=skylake,infiniband NodeName=gpu[01-04] Features=a100,nvlink Gres=gpu:a100:4 NodeName=gpu[05-08] Features=v100 Gres=gpu:v100:4 ``` --- ## Partition Definitions ```ini PartitionName=batch Nodes=node[001-048] Default=YES MaxTime=7-00:00:00 DefaultTime=01:00:00 State=UP PartitionName=gpu Nodes=gpu[01-08] MaxTime=2-00:00:00 DefaultTime=04:00:00 State=UP PartitionName=highmem Nodes=bigmem[1-4] MaxTime=7-00:00:00 State=UP PartitionName=interactive Nodes=node[001-048] MaxTime=4:00:00 DefaultTime=01:00:00 State=UP MaxNodes=1 PartitionName=debug Nodes=node[001-004] MaxTime=00:30:00 MaxNodes=2 State=UP PriorityTier=100 ``` Key partition parameters: | Parameter | Description | |-----------|-------------| | `Nodes` | Which nodes belong to this partition | | `Default` | Default partition for jobs without `-p` | | `MaxTime` | Maximum walltime allowed | | `DefaultTime` | Default walltime if user doesn't specify | | `MaxNodes` | Max nodes per job | | `PriorityTier` | Scheduling evaluation order (higher = first) | | `AllowGroups` | Restrict access to Unix groups | | `AllowAccounts` | Restrict access to Slurm accounts | | `QOS` | Attach a partition-level QOS for limits | | `OverSubscribe` | Allow node sharing (`YES`, `NO`, `FORCE`) | | `PreemptMode` | How preemption works in this partition | --- ## Applying Configuration Changes ```bash # After editing slurm.conf: scontrol reconfigure # Verify the change took effect scontrol show config | grep <parameter> ``` Some changes require a daemon restart rather than a reconfigure (notably changes to `SelectType`, `AuthType`, or node definitions that change hardware counts). --- ## scontrol: Runtime Administration `scontrol` is the primary admin command for viewing and modifying Slurm objects at runtime. ### View Configuration ```bash scontrol show config # All running parameters scontrol show config | grep Priority # Filter specific settings scontrol show partition # All partitions scontrol show node node001 # Specific node details scontrol show job 12345 # Specific job details ``` ### Modify Runtime State ```bash # Drain a node for maintenance scontrol update NodeName=node001 State=DRAIN Reason="memory replacement" # Return a node to service scontrol update NodeName=node001 State=RESUME # Change partition state scontrol update PartitionName=debug State=DOWN # Update a pending job scontrol update JobId=12345 TimeLimit=24:00:00 ``` ### Create a Reservation ```bash scontrol create reservation ReservationName=maint \ StartTime=now+2hours Duration=120 \ Nodes=node[001-004] User=root Flags=MAINT ``` --- ## sacctmgr: Account Administration `sacctmgr` manages the accounting database -- clusters, accounts, users, QOS, and resource limits. See the [Accounts & Fairshare](accounts-fairshare.md) module for detailed usage. ```bash # Quick examples sacctmgr show cluster sacctmgr show account sacctmgr show user sacctmgr show qos sacctmgr show association format=Account,User,Fairshare,MaxJobs,MaxTRES ``` --- ## Configuration Best Practices 1. **Version control your config files.** Put `slurm.conf`, `cgroup.conf`, `gres.conf` in git. 2. **Use DEFAULT directives** to reduce repetition in node definitions. 3. **Set `DefaultTime` on partitions.** Users who forget `--time` get a reasonable default instead of the partition maximum. 4. **Set `ReturnToService=2`** so nodes automatically rejoin after transient failures. 5. **Enable `cgroup/v2`** (or v1 on older kernels) for resource enforcement. Without it, memory limits are advisory only. 6. **Start with `SlurmctldDebug=info`** and increase only when troubleshooting. > **ParallelCluster Note:** Edit cluster configuration YAML, not slurm.conf directly. Use `CustomSlurmSettings` for parameters not exposed in the YAML. Some parameters are deny-listed.

Exercises

1. Find a specific configuration parameter with scontrol

Use scontrol show config to determine the current value of SchedulerParameters. Identify which backfill settings are active on the running cluster.

Hint / Solution

scontrol show config | grep SchedulerParameters
# Example output:
# SchedulerParameters = bf_max_job_test=5000,bf_interval=30,bf_resolution=600

# You can also check other parameters the same way:
scontrol show config | grep SelectType
scontrol show config | grep PriorityWeight

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ scontrol show config | grep SchedulerParameters
SchedulerParameters     = nohold_on_prolog_fail

1. Change the slurmctld debug level temporarily

Increase the controller debug level to debug2 for troubleshooting, verify it took effect, then set it back to info. Do this without editing slurm.conf or restarting any daemon.

Hint / Solution

# Increase debug level (takes effect immediately)
scontrol setdebug debug2

# Verify the change
scontrol show config | grep SlurmctldDebug
# Should show: SlurmctldDebug = debug2

# Watch logs briefly to confirm verbose output
tail -5 /var/log/slurm/slurmctld.log

# Reset to normal
scontrol setdebug info
scontrol show config | grep SlurmctldDebug

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sudo /opt/slurm/bin/scontrol setdebug debug2

$ scontrol show config | grep SlurmctldDebug
SlurmctldDebug          = debug2

$ scontrol show config | grep SlurmctldDebug
SlurmctldDebug          = info

1. Add a new node definition

Add a new high-memory node bigmem5 with 128 CPUs and 2 TB of RAM to slurm.conf, place it in the highmem partition, and apply the change. Verify the node appears in sinfo.

Hint / Solution

# Edit slurm.conf -- add to the node definitions section:
#   NodeName=bigmem5 CPUs=128 RealMemory=2048000
# Update the highmem partition line:
#   PartitionName=highmem Nodes=bigmem[1-4],bigmem5 ...

# Apply the change (adding a node requires a restart, not just reconfigure)
systemctl restart slurmctld

# On the new node, start slurmd
ssh bigmem5 systemctl start slurmd

# Verify
sinfo -n bigmem5
scontrol show node bigmem5

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ scontrol show partition val_extra
PartitionName=val_extra
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   Nodes=val_node1
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=1 TotalNodes=1 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=1,mem=128M,node=1,billing=1
   ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO

$ scontrol show node val_node1 || true
Node val_node1 not found

$ sudo grep '^NodeName=val_node1' /opt/slurm/etc/slurm.conf
NodeName=val_node1 CPUs=1 RealMemory=128 State=FUTURE

1. Reconfigure the cluster without a restart

Change DefaultTime on the batch partition from 01:00:00 to 02:00:00 in slurm.conf, then apply the change using scontrol reconfigure. Verify the new default is active.

Hint / Solution

# Edit slurm.conf -- change the batch partition line:
#   PartitionName=batch ... DefaultTime=02:00:00 ...

# Apply without restart
scontrol reconfigure

# Verify the change took effect
scontrol show partition batch | grep DefaultTime
# Should show: DefaultTime=02:00:00

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ scontrol show partition val_extra2
PartitionName=val_extra2
   AllowGroups=ALL AllowAccounts=ALL AllowQos=ALL
   AllocNodes=ALL Default=NO QoS=N/A
   DefaultTime=NONE DisableRootJobs=NO ExclusiveUser=NO ExclusiveTopo=NO GraceTime=0 Hidden=NO
   MaxNodes=UNLIMITED MaxTime=UNLIMITED MinNodes=0 LLN=NO MaxCPUsPerNode=UNLIMITED MaxCPUsPerSocket=UNLIMITED
   Nodes=val_node2
   PriorityJobFactor=1 PriorityTier=1 RootOnly=NO ReqResv=NO OverSubscribe=NO
   OverTimeLimit=NONE PreemptMode=OFF
   State=UP TotalCPUs=2 TotalNodes=1 SelectTypeParameters=NONE
   JobDefaults=(null)
   DefMemPerNode=UNLIMITED MaxMemPerNode=UNLIMITED
   TRES=cpu=2,mem=256M,node=1,billing=2
   ResumeTimeout=GLOBAL SuspendTimeout=GLOBAL SuspendTime=GLOBAL PowerDownOnIdle=NO

References

• SchedMD: slurm.conf man page
• SchedMD: cgroup.conf man page
• SchedMD: gres.conf man page
• SchedMD: scontrol man page
• SchedMD: Scheduling Configuration Guide