Installation

<!-- audience: L4 --> <!-- deployment: all --> <!-- sge-source: SGE Admin 02-Install --> # Installation > **ParallelCluster / PCS Note:** Slurm installation, daemon configuration, and database setup are handled automatically on both platforms. This module is most relevant for on-prem deployments. On ParallelCluster, use the cluster YAML config and `CustomSlurmSettings` for tuning. On PCS, configure through the PCS API. ## Prerequisites Before installing Slurm, verify these requirements across all nodes: ### Consistent Identity - **Uniform UID/GID mapping** across all nodes (LDAP, SSSD, NIS, or synced local accounts) - Create a dedicated `slurm` user (SlurmUser) with the same UID on all nodes - Munge user (if using Munge auth) with same UID on all nodes > **Lesson from SGE:** Grid Engine was "obscenely sensitive" to name resolution issues. Slurm is more tolerant but still requires consistent forward DNS and matching hostnames across the cluster. ### Shared Filesystem - Slurm configuration files should be accessible on all nodes (NFS, shared mount, or configless mode) - Job output directories must be accessible from compute nodes - `StateSaveLocation` should be on reliable local or shared storage ### Time Synchronization - NTP or chrony must be running on all nodes - Time skew causes authentication failures (Munge has a ~2 minute tolerance window) ### Network - TCP connectivity between all Slurm hosts on ports 6817-6819 - Firewall rules to allow slurmctld ↔ slurmd and slurmctld ↔ slurmdbd --- ## Authentication Setup Choose your authentication method before installing: ### Option A: Munge (Traditional) ```bash # Install on ALL nodes yum install munge munge-libs munge-devel # RHEL/Rocky apt install munge libmunge-dev # Ubuntu # Generate key on one host, distribute to all dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key chown munge:munge /etc/munge/munge.key chmod 400 /etc/munge/munge.key # Copy to all nodes (same key everywhere) scp /etc/munge/munge.key node001:/etc/munge/munge.key # ... repeat for all nodes # Start on all nodes systemctl enable --now munge # Test munge -n | unmunge # Should succeed munge -n | ssh node001 unmunge # Should succeed across nodes ``` --- ## Option B: auth/slurm (Simpler, 23.11+) ```bash # Generate a shared key dd if=/dev/urandom bs=1 count=1024 > /etc/slurm/slurm.key chown slurm:slurm /etc/slurm/slurm.key chmod 400 /etc/slurm/slurm.key # Distribute to all nodes # In slurm.conf: AuthType=auth/slurm ``` No separate daemon required -- Slurm daemons handle authentication internally. ## Option C: Configless with sackd (Login Nodes) For login nodes that don't need a full Slurm installation: ```bash # On login nodes, start sackd pointing to the controller sackd --conf-server mgmt01:6817 ``` sackd fetches configuration from slurmctld and provides authentication services. --- ## Package Installation ### RPM-based (RHEL, Rocky, AlmaLinux) ```bash # Management node yum install slurm slurm-slurmctld slurm-slurmdbd slurm-perlapi # Compute nodes yum install slurm slurm-slurmd # Submit/login nodes yum install slurm slurm-perlapi # or just sackd # Optional yum install slurm-slurmrestd slurm-torque # REST API, PBS compatibility ``` ### DEB-based (Ubuntu, Debian) ```bash # Management node apt install slurmctld slurmdbd slurm-client # Compute nodes apt install slurmd slurm-client # Submit/login nodes apt install slurm-client ``` --- ## Database Setup (SlurmDBD) SlurmDBD requires MySQL or MariaDB: ```bash # Install MariaDB yum install mariadb-server # RHEL/Rocky apt install mariadb-server # Ubuntu # Start and secure systemctl enable --now mariadb mysql_secure_installation # Create Slurm database and user mysql -u root -p <<EOF CREATE DATABASE slurm_acct_db; CREATE USER 'slurm'@'localhost' IDENTIFIED BY 'secure_password'; GRANT ALL ON slurm_acct_db.* TO 'slurm'@'localhost'; FLUSH PRIVILEGES; EOF ``` --- ## slurmdbd.conf ```ini # /etc/slurm/slurmdbd.conf (chmod 600, owned by slurm) AuthType=auth/munge # or auth/slurm DbdHost=localhost StorageHost=localhost StoragePort=3306 StorageLoc=slurm_acct_db StorageUser=slurm StoragePass=secure_password StorageType=accounting_storage/mysql LogFile=/var/log/slurm/slurmdbd.log PidFile=/run/slurmdbd.pid SlurmUser=slurm ``` ```bash systemctl enable --now slurmdbd ``` --- ## Basic slurm.conf A minimal working configuration: ```ini # /etc/slurm/slurm.conf ClusterName=mycluster SlurmctldHost=mgmt01 # Authentication AuthType=auth/munge # or auth/slurm # Scheduling SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory # Accounting AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=mgmt01 JobAcctGatherType=jobacct_gather/cgroup # Process tracking and task management ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity # Logging SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log SlurmctldDebug=info SlurmdDebug=info # State StateSaveLocation=/var/spool/slurmctld SlurmdSpoolDir=/var/spool/slurmd # Timeouts SlurmctldTimeout=120 SlurmdTimeout=300 InactiveLimit=0 MinJobAge=300 KillWait=30 WaitTime=0 # Priority PriorityType=priority/multifactor PriorityWeightFairshare=100000 PriorityWeightAge=1000 PriorityWeightPartition=10000 PriorityWeightJobSize=1000 PriorityDecayHalfLife=14-0 ``` --- ## Node and Partition Definitions Continuing slurm.conf with hardware and partition layout: ```ini # Node definitions GresTypes=gpu NodeName=DEFAULT CPUs=64 RealMemory=256000 State=UNKNOWN NodeName=node[001-048] NodeAddr=node[001-048] NodeName=gpu[01-08] Gres=gpu:a100:4 CPUs=64 RealMemory=512000 # Partition definitions PartitionName=batch Nodes=node[001-048] Default=YES MaxTime=7-00:00:00 State=UP PartitionName=gpu Nodes=gpu[01-08] MaxTime=2-00:00:00 State=UP PartitionName=interactive Nodes=node[001-048] MaxTime=4:00:00 State=UP ``` --- ## cgroup.conf Essential for resource enforcement: ```ini # /etc/slurm/cgroup.conf CgroupPlugin=cgroup/v2 # or cgroup/v1 for older systems ConstrainCores=yes ConstrainRAMSpace=yes ConstrainSwapSpace=yes ConstrainDevices=yes # Required for GPU isolation ``` --- ## Starting Services ```bash # Management node systemctl enable --now slurmctld # Compute nodes (all) systemctl enable --now slurmd # Verify sinfo # Should show partitions and nodes scontrol show nodes # Should list all nodes ``` --- ## Register Cluster in Accounting ```bash # Add the cluster to SlurmDBD sacctmgr add cluster mycluster # Create a root account sacctmgr add account root # Add initial user sacctmgr add user admin account=root adminlevel=admin ``` --- ## Verification Checklist ```bash # 1. Auth works srun hostname # Should run on a compute node # 2. Accounting works sacctmgr show cluster # Should list your cluster sacctmgr show account # Should list accounts # 3. Job submission works sbatch --wrap="hostname; sleep 30" squeue # Should show the job # 4. Node health sinfo # All nodes should be idle/available scontrol show node node001 # Verify resources reported correctly ``` --- ## Deployment-Specific Notes > **ParallelCluster:** Installation is handled by ParallelCluster. You configure Slurm through the cluster YAML config file, not by editing slurm.conf directly. Use `CustomSlurmSettings` for advanced tuning. Some parameters are deny-listed. > **AWS PCS:** Slurm is fully managed. Focus on configuring compute resources, partitions, and networking through the PCS API.

Exercises

1. Verify Munge authentication across nodes

From the management node, generate a Munge credential and validate it on a remote compute node. Confirm that the credential encodes and decodes successfully on both sides.

Hint / Solution

# Local test
munge -n | unmunge

# Remote test -- pipe a credential to a compute node
munge -n | ssh cpu001 unmunge

# If the remote test fails, check:
# - Is the munge.key identical on both hosts?
# - Is the munge service running on cpu001?
# - Is the clock skew between hosts less than 2 minutes?
systemctl status munge
ssh cpu001 systemctl status munge

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ munge -n | unmunge | head -5
STATUS:          Success (0)
ENCODE_HOST:     ip-172-31-94-160.us-east-2.compute.internal (172.31.94.160)
ENCODE_TIME:     2026-04-15 02:00:40 +0000 (1776218440)
DECODE_TIME:     2026-04-15 02:00:40 +0000 (1776218440)
TTL:             300

1. Check SlurmDBD connectivity

Verify that slurmctld can communicate with slurmdbd. Look at the slurmdbd log to confirm the connection is established and the database is reachable.

Hint / Solution

# Check slurmdbd is running
systemctl status slurmdbd

# Verify the database connection from Slurm's perspective
sacctmgr show cluster
# If this returns your cluster name, slurmdbd is reachable

# Check slurmdbd log for connection messages
tail -50 /var/log/slurm/slurmdbd.log | grep -i "connection\|error\|database"

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ systemctl is-active slurmdbd
active

$ sacctmgr -nP show cluster format=Cluster,ControlHost,ControlPort,RPC
slurm-training-val|172.31.94.160|6820|11264

1. Verify the cluster is registered in accounting

Confirm that your cluster appears in the SlurmDBD accounting database. Then verify that a root account exists and at least one user is associated with it.

Hint / Solution

# List registered clusters
sacctmgr show cluster format=Cluster,ControlHost,ControlPort,RPC

# List accounts
sacctmgr show account format=Account,Description,Organization

# List user associations
sacctmgr show association format=Cluster,Account,User,Fairshare

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sacctmgr show cluster format=Cluster,ControlHost,ControlPort,RPC
   Cluster     ControlHost  ControlPort   RPC 
---------- --------------- ------------ ----- 
slurm-tra+   172.31.94.160         6820 11264 

$ sacctmgr show account format=Account,Description,Organization
   Account                Descr                  Org 
---------- -------------------- -------------------- 
 pcdefault parallelcluster def+                 none 
  research             research              bioteam 
      root default root account                 root 
  training             training              bioteam 

$ sacctmgr show association format=Cluster,Account,User,Fairshare | head -15
   Cluster    Account       User     Share 
---------- ---------- ---------- --------- 
slurm-tra+       root                    1 
slurm-tra+       root       root         1 
slurm-tra+  pcdefault                    1 
slurm-tra+  pcdefault   ec2-user         1 
slurm-tra+  pcdefault      slurm         1 
slurm-tra+   research                   50 
slurm-tra+   research  testuser1         1 
slurm-tra+   training                  100 

1. Test configless mode with sackd

On a login node (or a test node), start sackd pointing at the controller and confirm that it pulls down the Slurm configuration. Verify that Slurm client commands work without a local copy of slurm.conf.

Hint / Solution

# On the login/test node, remove or rename any local slurm.conf
mv /etc/slurm/slurm.conf /etc/slurm/slurm.conf.bak

# Start sackd pointing at the controller
sackd --conf-server mgmt01:6817

# Test that client commands work (they fetch config from sackd)
sinfo
squeue
scontrol show config | head -5

# If successful, sackd fetched the config from slurmctld
# Restore slurm.conf if you're not keeping configless mode
mv /etc/slurm/slurm.conf.bak /etc/slurm/slurm.conf

References

• SchedMD: Quick Start Administrator Guide
• SchedMD: slurm.conf man page
• SchedMD: slurmdbd.conf man page
• SchedMD: cgroup.conf man page
• SchedMD: Accounting and Resource Limits
• SchedMD: "Configless" Slurm
• MUNGE Authentication