Installation

# Installation

> **ParallelCluster / PCS Note:** Slurm installation, daemon configuration, and database setup are handled automatically on both platforms. This module is most relevant for on-prem deployments. On ParallelCluster, use the cluster YAML config and `CustomSlurmSettings` for tuning. On PCS, configure through the PCS API.

## Prerequisites

Before installing Slurm, verify these requirements across all nodes:

### Consistent Identity

- **Uniform UID/GID mapping** across all nodes (LDAP, SSSD, NIS, or synced local accounts)
- Create a dedicated `slurm` user (SlurmUser) with the same UID on all nodes
- Munge user (if using Munge auth) with same UID on all nodes

> **Lesson from SGE:** Grid Engine was "obscenely sensitive" to name resolution issues. Slurm is more tolerant but still requires consistent forward DNS and matching hostnames across the cluster.

### Shared Filesystem

- Slurm configuration files should be accessible on all nodes (NFS, shared mount, or configless mode)
- Job output directories must be accessible from compute nodes
- `StateSaveLocation` should be on reliable local or shared storage

### Time Synchronization

- NTP or chrony must be running on all nodes
- Time skew causes authentication failures (Munge has a ~2 minute tolerance window)

### Network

- TCP connectivity between all Slurm hosts on ports 6817-6819
- Firewall rules to allow slurmctld ↔ slurmd and slurmctld ↔ slurmdbd

---

## Authentication Setup

Choose your authentication method before installing:

### Option A: Munge (Traditional)

```bash
# Install on ALL nodes
yum install munge munge-libs munge-devel    # RHEL/Rocky
apt install munge libmunge-dev              # Ubuntu

# Generate key on one host, distribute to all
dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key
chown munge:munge /etc/munge/munge.key
chmod 400 /etc/munge/munge.key

# Copy to all nodes (same key everywhere)
scp /etc/munge/munge.key node001:/etc/munge/munge.key
# ... repeat for all nodes

# Start on all nodes
systemctl enable --now munge

# Test
munge -n | unmunge    # Should succeed
munge -n | ssh node001 unmunge    # Should succeed across nodes
```

### Option B: auth/slurm (Simpler, 23.11+)

```bash
# Generate a shared key
dd if=/dev/urandom bs=1 count=1024 > /etc/slurm/slurm.key
chown slurm:slurm /etc/slurm/slurm.key
chmod 400 /etc/slurm/slurm.key

# Distribute to all nodes
# In slurm.conf:
AuthType=auth/slurm
```

No separate daemon required -- Slurm daemons handle authentication internally.

### Option C: Configless with sackd (Login Nodes)

For login nodes that don't need a full Slurm installation:

```bash
# On login nodes, start sackd pointing to the controller
sackd --conf-server mgmt01:6817
```

sackd fetches configuration from slurmctld and provides authentication services.

---

## Package Installation

### RPM-based (RHEL, Rocky, AlmaLinux)

```bash
# Management node
yum install slurm slurm-slurmctld slurm-slurmdbd slurm-perlapi

# Compute nodes
yum install slurm slurm-slurmd

# Submit/login nodes
yum install slurm slurm-perlapi    # or just sackd

# Optional
yum install slurm-slurmrestd slurm-torque    # REST API, PBS compatibility
```

### DEB-based (Ubuntu, Debian)

```bash
# Management node
apt install slurmctld slurmdbd slurm-client

# Compute nodes
apt install slurmd slurm-client

# Submit/login nodes
apt install slurm-client
```

---

## Database Setup (SlurmDBD)

SlurmDBD requires MySQL or MariaDB:

```bash
# Install MariaDB
yum install mariadb-server    # RHEL/Rocky
apt install mariadb-server    # Ubuntu

# Start and secure
systemctl enable --now mariadb
mysql_secure_installation

# Create Slurm database and user
mysql -u root -p <<EOF
CREATE DATABASE slurm_acct_db;
CREATE USER 'slurm'@'localhost' IDENTIFIED BY 'secure_password';
GRANT ALL ON slurm_acct_db.* TO 'slurm'@'localhost';
FLUSH PRIVILEGES;
EOF
```

### slurmdbd.conf

```ini
# /etc/slurm/slurmdbd.conf (chmod 600, owned by slurm)
AuthType=auth/munge        # or auth/slurm
DbdHost=localhost
StorageHost=localhost
StoragePort=3306
StorageLoc=slurm_acct_db
StorageUser=slurm
StoragePass=secure_password
StorageType=accounting_storage/mysql
LogFile=/var/log/slurm/slurmdbd.log
PidFile=/run/slurmdbd.pid
SlurmUser=slurm
```

```bash
systemctl enable --now slurmdbd
```

---

## Basic slurm.conf

A minimal working configuration:

```ini
# /etc/slurm/slurm.conf
ClusterName=mycluster
SlurmctldHost=mgmt01

# Authentication
AuthType=auth/munge            # or auth/slurm

# Scheduling
SchedulerType=sched/backfill
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory

# Accounting
AccountingStorageType=accounting_storage/slurmdbd
AccountingStorageHost=mgmt01
JobAcctGatherType=jobacct_gather/cgroup

# Process tracking and task management
ProctrackType=proctrack/cgroup
TaskPlugin=task/cgroup,task/affinity

# Logging
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdLogFile=/var/log/slurm/slurmd.log
SlurmctldDebug=info
SlurmdDebug=info

# State
StateSaveLocation=/var/spool/slurmctld
SlurmdSpoolDir=/var/spool/slurmd

# Timeouts
SlurmctldTimeout=120
SlurmdTimeout=300
InactiveLimit=0
MinJobAge=300
KillWait=30
WaitTime=0

# Priority
PriorityType=priority/multifactor
PriorityWeightFairshare=100000
PriorityWeightAge=1000
PriorityWeightPartition=10000
PriorityWeightJobSize=1000
PriorityDecayHalfLife=14-0

# Node definitions
GresTypes=gpu
NodeName=DEFAULT CPUs=64 RealMemory=256000 State=UNKNOWN
NodeName=node[001-048] NodeAddr=node[001-048]
NodeName=gpu[01-08] Gres=gpu:a100:4 CPUs=64 RealMemory=512000

# Partition definitions
PartitionName=batch Nodes=node[001-048] Default=YES MaxTime=7-00:00:00 State=UP
PartitionName=gpu Nodes=gpu[01-08] MaxTime=2-00:00:00 State=UP
PartitionName=interactive Nodes=node[001-048] MaxTime=4:00:00 State=UP
```

---

## cgroup.conf

Essential for resource enforcement:

```ini
# /etc/slurm/cgroup.conf
CgroupPlugin=cgroup/v2         # or cgroup/v1 for older systems
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=yes            # Required for GPU isolation
```

---

## Starting Services

```bash
# Management node
systemctl enable --now slurmctld

# Compute nodes (all)
systemctl enable --now slurmd

# Verify
sinfo        # Should show partitions and nodes
scontrol show nodes    # Should list all nodes
```

---

## Register Cluster in Accounting

```bash
# Add the cluster to SlurmDBD
sacctmgr add cluster mycluster

# Create a root account
sacctmgr add account root

# Add initial user
sacctmgr add user admin account=root adminlevel=admin
```

---

## Verification Checklist

```bash
# 1. Auth works
srun hostname                    # Should run on a compute node

# 2. Accounting works
sacctmgr show cluster            # Should list your cluster
sacctmgr show account            # Should list accounts

# 3. Job submission works
sbatch --wrap="hostname; sleep 30"
squeue                           # Should show the job

# 4. Node health
sinfo                            # All nodes should be idle/available
scontrol show node node001       # Verify resources reported correctly
```

---

## Deployment-Specific Notes

> **ParallelCluster:** Installation is handled by ParallelCluster. You configure Slurm through the cluster YAML config file, not by editing slurm.conf directly. Use `CustomSlurmSettings` for advanced tuning. Some parameters are deny-listed.

> **AWS PCS:** Slurm is fully managed. Focus on configuring compute resources, partitions, and networking through the PCS API.

Exercises¶

Verify Munge authentication across nodes

From the management node, generate a Munge credential and validate it on a remote compute node. Confirm that the credential encodes and decodes successfully on both sides.

Hint / Solution

# Local test
munge -n | unmunge

# Remote test -- pipe a credential to a compute node
munge -n | ssh cpu001 unmunge

# If the remote test fails, check:
# - Is the munge.key identical on both hosts?
# - Is the munge service running on cpu001?
# - Is the clock skew between hosts less than 2 minutes?
systemctl status munge
ssh cpu001 systemctl status munge

Check SlurmDBD connectivity

Verify that slurmctld can communicate with slurmdbd. Look at the slurmdbd log to confirm the connection is established and the database is reachable.

Hint / Solution

# Check slurmdbd is running
systemctl status slurmdbd

# Verify the database connection from Slurm's perspective
sacctmgr show cluster
# If this returns your cluster name, slurmdbd is reachable

# Check slurmdbd log for connection messages
tail -50 /var/log/slurm/slurmdbd.log | grep -i "connection\|error\|database"

Verify the cluster is registered in accounting

Confirm that your cluster appears in the SlurmDBD accounting database. Then verify that a root account exists and at least one user is associated with it.

Hint / Solution

# List registered clusters
sacctmgr show cluster format=Cluster,ControlHost,ControlPort,RPC

# List accounts
sacctmgr show account format=Account,Description,Organization

# List user associations
sacctmgr show association format=Cluster,Account,User,Fairshare

Test configless mode with sackd

On a login node (or a test node), start sackd pointing at the controller and confirm that it pulls down the Slurm configuration. Verify that Slurm client commands work without a local copy of slurm.conf.

Hint / Solution

# On the login/test node, remove or rename any local slurm.conf
mv /etc/slurm/slurm.conf /etc/slurm/slurm.conf.bak

# Start sackd pointing at the controller
sackd --conf-server mgmt01:6817

# Test that client commands work (they fetch config from sackd)
sinfo
squeue
scontrol show config | head -5

# If successful, sackd fetched the config from slurmctld
# Restore slurm.conf if you're not keeping configless mode
mv /etc/slurm/slurm.conf.bak /etc/slurm/slurm.conf

Installation

Exercises¶

References¶