On-Premise

<!-- audience: L4 --> <!-- deployment: on-prem --> <!-- sge-source: SGE Admin 02-Install + 03-Config --> # On-Premise Slurm Deployment ## Overview This module covers deploying Slurm on traditional on-premise infrastructure -- bare-metal servers, rack-mounted GPUs, and institutional HPC clusters. On-prem Slurm gives you full control over hardware, networking, storage, and configuration, but requires careful planning of every layer in the stack. The admin track modules (00-10) cover Slurm concepts, commands, and configuration in detail. This module focuses on the infrastructure decisions and deployment patterns specific to on-premise environments. --- ## Infrastructure Planning ### Node Roles A production Slurm cluster requires several distinct node roles: | Role | Count | Purpose | |------|-------|---------| | **Management node(s)** | 1-2 | slurmctld (primary + optional backup for HA) | | **Database host** | 1 | slurmdbd + MySQL/MariaDB | | **Login/submit nodes** | 1+ | User access, job submission, no compute | | **Compute nodes** | N | slurmd, where jobs actually run | | **GPU nodes** | 0+ | Compute nodes with GPUs (RELION, AlphaFold, Desmond) | | **Storage servers** | 1+ | Shared filesystems (NFS, Lustre, BeeGFS, GPFS) | For small clusters (under ~20 nodes), the management node and database host can share a single server. For larger clusters, separate them. ### Hardware Sizing Guidelines **Management node:** - 4-8 CPU cores, 16-64 GB RAM (scales with cluster size) - Fast local SSD for `StateSaveLocation` (state checkpoints) - For clusters over 1,000 nodes, dedicate 8+ cores and 64+ GB RAM **Database host:** - 4-8 CPU cores, 32+ GB RAM - Fast local SSD for MySQL data directory - Sufficient IOPS for accounting write load **Login nodes:** - Sized to handle concurrent user sessions (compilation, data staging) - Same OS, libraries, and module system as compute nodes - No slurmd -- users should not run compute here **Compute nodes:** - Sized for your workload (CPU-bound genomics vs. GPU-bound cryo-EM) - RealMemory in slurm.conf should reflect usable RAM (total minus OS overhead) - Local scratch (SSD/NVMe) for temporary job data improves I/O-heavy workloads ### Example Life Science Cluster Layout ``` Management: mgmt01, mgmt02 (HA pair) Database: db01 (slurmdbd + MariaDB) Login: login01, login02 (user access) Compute: cpu[001-064] (64-core, 256 GB RAM -- genomics, MD) High-memory: bigmem[1-4] (128-core, 1 TB RAM -- genome assembly) GPU: gpu[01-08] (64-core, 512 GB RAM, 4x A100 -- cryo-EM, ML) Storage: Lustre OSS/MDS or BeeGFS (shared /data, /scratch) ``` --- ## Network Architecture ### Network Requirements Slurm uses TCP for all daemon communication: | Communication Path | Default Port | Purpose | |--------------------|-------------|---------| | slurmctld | 6817 | Controller RPC | | slurmd | 6818 | Node daemon RPC | | slurmdbd | 6819 | Accounting database daemon | | slurmrestd | 6820 | REST API (optional) | | sackd | 6817 | Authentication on login nodes (configless) | **Firewall rules required:** - All compute nodes → slurmctld (TCP 6817) - slurmctld → all compute nodes (TCP 6818) - slurmctld → slurmdbd (TCP 6819) - Login nodes → slurmctld (TCP 6817) ### Network Topology Most life science clusters use a flat Ethernet fabric: ``` ┌──────────────┐ │ Management │ │ Network │──── mgmt01, mgmt02, db01 │ (1 GbE) │ └──────┬───────┘ │ ┌──────────────┼──────────────┐ │ │ │ ┌─────▼─────┐ ┌─────▼─────┐ ┌─────▼──────┐ │ Compute │ │ Compute │ │ Storage │ │ Network │ │ Network │ │ Network │ │ (25 GbE) │ │ (HDR IB) │ │ (25+ GbE) │ └────────────┘ └────────────┘ └─────────────┘ cpu[001-064] gpu[01-08] Lustre OSS/MDS ``` **Recommendations:** - Separate management and compute traffic (VLANs or physical networks) - InfiniBand for MPI-heavy workloads (molecular dynamics, quantum chemistry) - High-bandwidth Ethernet (25+ GbE) for storage traffic - Dedicated storage network if running a parallel filesystem ### Topology-Aware Scheduling For large clusters, configure Slurm's topology plugin to prefer co-located nodes: ```ini # slurm.conf TopologyPlugin=topology/tree ``` ```ini # topology.conf SwitchName=leaf01 Nodes=cpu[001-032] SwitchName=leaf02 Nodes=cpu[033-064] SwitchName=spine01 Switches=leaf01,leaf02 ``` This helps Slurm place MPI jobs on nodes sharing the same switch, reducing inter-switch hops. --- ## Shared Filesystem Every Slurm cluster needs shared storage accessible from all login and compute nodes. The filesystem choice significantly impacts performance. ### Filesystem Options | Filesystem | Type | Best For | Typical Scale | |------------|------|----------|--------------| | **NFS** | Network file server | Small clusters (<50 nodes), home dirs | 1-10 TB | | **Lustre** | Parallel filesystem | Large I/O, cryo-EM datasets, genomics | 50 TB - PB | | **BeeGFS** | Parallel filesystem | Mid-size clusters, ease of management | 10 TB - PB | | **GPFS/Spectrum Scale** | Parallel filesystem | Enterprise, mixed workloads | 50 TB - PB | | **WekaFS** | Flash-optimized parallel | AI/ML, GPU-heavy workloads | 10 TB - PB | ### Storage Layout A common pattern for life science HPC: ``` /home/ NFS, user home directories, quotas enforced /software/ NFS or shared mount, application installations + modules /data/ Parallel filesystem (Lustre/BeeGFS), persistent project data /scratch/ Parallel filesystem, temporary job I/O, auto-purged /tmp or /local/ Local SSD on each compute node, per-job scratch ``` ### Slurm and Shared Storage Key configuration considerations: ```ini # slurm.conf # Job output defaults to the submission directory -- must be on shared storage # or use --output with a shared path # StateSaveLocation should be on fast LOCAL storage (not NFS) # unless running HA (then shared between controllers only) StateSaveLocation=/var/spool/slurmctld # Spool directory for slurmd -- local to each compute node SlurmdSpoolDir=/var/spool/slurmd # Temporary filesystem for job prolog/epilog TmpFS=/local/tmp ``` ### Prolog/Epilog for Local Scratch Create per-job scratch directories on local SSDs: ```bash #!/bin/bash # /etc/slurm/prolog.sh -- runs as root before each job mkdir -p /local/tmp/slurm-${SLURM_JOB_ID} chown ${SLURM_JOB_USER}:${SLURM_JOB_USER} /local/tmp/slurm-${SLURM_JOB_ID} ``` ```bash #!/bin/bash # /etc/slurm/epilog.sh -- runs as root after each job rm -rf /local/tmp/slurm-${SLURM_JOB_ID} ``` ```ini # slurm.conf Prolog=/etc/slurm/prolog.sh Epilog=/etc/slurm/epilog.sh PrologFlags=Alloc # Run once per allocation, not per task ``` --- ## Identity and Authentication ### User Identity Slurm requires consistent UID/GID mapping across all nodes: - **LDAP/SSSD** -- preferred for institutional clusters, integrates with Active Directory - **FreeIPA** -- open-source identity management with Kerberos - **NIS/NIS+** -- legacy option, still common in academic HPC - **Synced local accounts** -- small clusters only, fragile at scale The `slurm` service account (SlurmUser) must have the same UID on every node: ```bash # On ALL nodes (or via LDAP) useradd -r -u 64030 -s /sbin/nologin slurm ``` ### Slurm Authentication Choose one authentication method for the cluster: **Option A: Munge (traditional, default)** - Symmetric key shared across all nodes - Simple and battle-tested - Requires time synchronization (NTP/chrony, ±2 minute tolerance) **Option B: auth/slurm (23.11+, simpler alternative)** - Uses `slurm.key` managed by slurmctld - No additional daemon to install - Compute nodes get the key via configless mode or file distribution **Option C: auth/jwt (for REST API and external identity)** - JSON Web Tokens for programmatic access - Pairs with auth/munge or auth/slurm as the primary AuthType - Integrates with KeyCloak, Okta, AWS Cognito for SSO See the [Installation](../admin/01-installation.md) module for detailed setup steps. ### nss_slurm for Compute Nodes For clusters where compute nodes lack direct LDAP access, nss_slurm passes user identity from slurmctld to compute nodes via job credentials: ```ini # slurm.conf LaunchParameters=enable_nss_slurm AuthInfo=use_client_ids ``` This allows compute nodes to operate without LDAP/SSSD, reducing external dependencies. --- ## Configless Mode Configless Slurm eliminates the need to distribute configuration files to every node. Compute nodes and login nodes pull config directly from slurmctld. ### Setup ```ini # slurm.conf (on the controller) SlurmctldParameters=enable_configless ``` **Compute nodes** -- start slurmd with `--conf-server`: ```bash slurmd --conf-server slurmctl-primary:6817,slurmctl-backup:6817 ``` Or use DNS SRV records (preferred): ``` _slurmctld._tcp 3600 IN SRV 0 0 6817 slurmctl-primary _slurmctld._tcp 3600 IN SRV 10 0 6817 slurmctl-backup ``` **Login nodes** -- run `sackd` to cache config and provide authentication: ```bash # sackd caches config locally, reducing RPC load on slurmctld systemctl enable --now sackd ``` sackd is recommended on login nodes with many concurrent users, as raw configless mode generates an RPC to slurmctld for every client command. ### Multi-Cluster Login Nodes (Slurm 25.05+) Starting with Slurm 25.05, multiple sackd daemons can co-exist on the same login node. Each daemon connects to a different cluster's slurmctld, allowing users to access multiple Slurm clusters from a single machine. The key requirement is that each sackd instance uses a separate `RuntimeDirectory` in its systemd service file. Users select which cluster to interact with by setting `SLURM_CONF` to point at the appropriate cluster's cached configuration. **Example: two sackd services on one login node** ```ini # /etc/systemd/system/sackd-clusterA.service [Unit] Description=Slurm auth and cred daemon for clusterA After=network-online.target [Service] Type=notify ExecStart=/usr/sbin/sackd --conf-server slurmctld-a:6817 RuntimeDirectory=sackd-clusterA Environment=SLURM_CONF=/run/sackd-clusterA/conf/slurm.conf [Install] WantedBy=multi-user.target ``` ```ini # /etc/systemd/system/sackd-clusterB.service [Unit] Description=Slurm auth and cred daemon for clusterB After=network-online.target [Service] Type=notify ExecStart=/usr/sbin/sackd --conf-server slurmctld-b:6817 RuntimeDirectory=sackd-clusterB Environment=SLURM_CONF=/run/sackd-clusterB/conf/slurm.conf [Install] WantedBy=multi-user.target ``` Users activate a cluster by pointing `SLURM_CONF` at the right config: ```bash # Work with clusterA export SLURM_CONF=/run/sackd-clusterA/conf/slurm.conf sinfo # shows clusterA partitions # Switch to clusterB export SLURM_CONF=/run/sackd-clusterB/conf/slurm.conf sinfo # shows clusterB partitions ``` This is useful in environments with multiple Slurm clusters (for example, separate clusters for different departments or security domains) where a shared login node simplifies user access. ### Benefits - Single source of truth for configuration - No need to push slurm.conf to every node - New nodes join the cluster with zero config files - Config changes take effect on the next slurmd restart or `scontrol reconfigure` --- ## Installation Walkthrough This is a condensed walkthrough. See the [Installation](../admin/01-installation.md) module for full details. ### 1. Install Packages ```bash # RHEL/Rocky 8+ dnf install slurm slurm-slurmd slurm-slurmctld slurm-slurmdbd \ slurm-perlapi slurm-torque munge munge-libs mariadb-server # Ubuntu 22.04+ apt install slurmd slurmctld slurmdbd slurm-client \ munge libmunge-dev mariadb-server ``` ### 2. Configure Authentication ```bash # Munge (traditional) dd if=/dev/urandom bs=1 count=1024 > /etc/munge/munge.key chmod 400 /etc/munge/munge.key chown munge:munge /etc/munge/munge.key # Distribute to all nodes, then: systemctl enable --now munge ``` ### 3. Configure MariaDB for slurmdbd ```bash systemctl enable --now mariadb mysql -u root <<EOF CREATE DATABASE slurm_acct_db; CREATE USER 'slurm'@'localhost' IDENTIFIED BY 'your-secure-password'; GRANT ALL ON slurm_acct_db.* TO 'slurm'@'localhost'; FLUSH PRIVILEGES; EOF ``` ### 4. Create slurm.conf Use the [SchedMD Configuration Tool](https://slurm.schedmd.com/configurator.html) as a starting point, then customize: ```ini # slurm.conf -- minimal on-prem example ClusterName=biocluster SlurmctldHost=mgmt01(10.0.1.1) SlurmctldHost=mgmt02(10.0.1.2) # Authentication AuthType=auth/munge # Scheduling SchedulerType=sched/backfill SelectType=select/cons_tres SelectTypeParameters=CR_Core_Memory # Accounting AccountingStorageType=accounting_storage/slurmdbd AccountingStorageHost=db01 JobAcctGatherType=jobacct_gather/cgroup # Logging SlurmctldLogFile=/var/log/slurm/slurmctld.log SlurmdLogFile=/var/log/slurm/slurmd.log # State StateSaveLocation=/var/spool/slurmctld SlurmdSpoolDir=/var/spool/slurmd # Resource enforcement ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity # Nodes NodeName=DEFAULT CPUs=64 Boards=1 SocketsPerBoard=2 CoresPerSocket=16 ThreadsPerCore=2 RealMemory=256000 State=UNKNOWN NodeName=cpu[001-064] NodeName=bigmem[1-4] CPUs=128 RealMemory=1024000 NodeName=gpu[01-08] CPUs=64 RealMemory=512000 Gres=gpu:a100:4 # Partitions PartitionName=DEFAULT MaxTime=7-00:00:00 DefaultTime=01:00:00 State=UP PartitionName=batch Nodes=cpu[001-064] Default=YES PartitionName=highmem Nodes=bigmem[1-4] PartitionName=gpu Nodes=gpu[01-08] ``` ### 5. Start Services (in order) ```bash # 1. slurmdbd first (on db01) systemctl enable --now slurmdbd # 2. slurmctld (on mgmt01) systemctl enable --now slurmctld # 3. slurmd (on all compute nodes) systemctl enable --now slurmd # 4. Verify sinfo # PARTITION AVAIL TIMELIMIT NODES STATE NODELIST # batch* up 7-00:00:00 64 idle cpu[001-064] # highmem up 7-00:00:00 4 idle bigmem[1-4] # gpu up 7-00:00:00 8 idle gpu[01-08] ``` ### Why Accounting Must Be Enabled > **Best Practice:** Always deploy slurmdbd and enable accounting storage (`AccountingStorageType=accounting_storage/slurmdbd`), even on small clusters. Accounting is not optional overhead -- it is the foundation for most of Slurm's advanced scheduling and resource management capabilities. Without accounting, the following features are **unavailable**: > > - **Fairshare scheduling** (priority/multifactor) -- cannot weight user/group priority by historical usage > - **QOS enforcement** -- cannot set resource limits per user, account, or group > - **Job history and reporting** (`sacct`, `sreport`) -- no post-job resource usage data > - **License-aware scheduling** -- cannot integrate FLEXlm license counts (required for Schrodinger and other commercial software) > - **Account-based usage tracking and chargeback** -- no way to attribute compute hours to departments or grants > - **Resource limit enforcement** (MaxJobs, MaxSubmitJobs, GrpTRES) -- cannot prevent any single user from consuming the entire cluster > > The slurm.conf example in Step 4 above already includes `AccountingStorageType=accounting_storage/slurmdbd`. If you skip slurmdbd setup, Slurm will still start, but you will be running a cluster with no scheduling intelligence beyond basic FIFO/backfill. This is one of the most common mistakes in new Slurm deployments. ### 6. Register the Cluster in Accounting ```bash sacctmgr add cluster biocluster sacctmgr add account research Description="Research groups" sacctmgr add user jsmith Account=research ``` --- ## Large Cluster Tuning For clusters with 1,000+ nodes, tune OS and Slurm parameters: ### OS-Level Tuning ```bash # /etc/sysctl.conf on management nodes fs.file-max = 388067 net.ipv4.tcp_max_syn_backlog = 4096 net.core.somaxconn = 4096 ``` ### Slurm Scheduler Tuning ```ini # slurm.conf SchedulerParameters=bf_max_job_test=5000,bf_interval=30,bf_resolution=600 # Reduce slurmctld RPC load MessageTimeout=30 BatchStartTimeout=30 SlurmctldParameters=enable_configless # Enable configless to eliminate config distribution # Use sackd on login nodes to reduce per-command RPCs ``` ### Job Throughput ```ini # For high job submission rates (>100 jobs/sec) MaxJobCount=300000 MinJobAge=300 ``` --- ## Configuration Management ### Version Control Keep all Slurm config files in a version control system (Git): ``` slurm-config/ ├── slurm.conf ├── slurmdbd.conf ├── cgroup.conf ├── gres.conf ├── topology.conf ├── prolog.sh ├── epilog.sh └── health_check.sh ``` ### Config Distribution Three approaches (choose one): | Method | Pros | Cons | |--------|------|------| | **Configless mode** | Zero-touch, single source of truth | Requires 20.02+, sackd recommended | | **Shared filesystem** | Simple, immediate changes | NFS dependency | | **Config management (Ansible/Puppet)** | Full control, auditable | Requires config mgmt infrastructure | ### Rolling Changes ```bash # After editing slurm.conf: # 1. Test syntax slurmd -C # Print node config (validates parsing) # 2. Push config (if not using configless) ansible all -m copy -a "src=slurm.conf dest=/etc/slurm/slurm.conf" # 3. Reconfigure live scontrol reconfigure # 4. Verify scontrol show config | grep -i "changed_parameter" ``` --- ## Monitoring and Health Checks ### Node Health Checks ```ini # slurm.conf HealthCheckProgram=/etc/slurm/health_check.sh HealthCheckInterval=600 HealthCheckNodeState=ANY ``` Example health check for a life science cluster: ```bash #!/bin/bash # /etc/slurm/health_check.sh # GPU health (critical for cryo-EM and ML nodes) if command -v nvidia-smi &>/dev/null; then nvidia-smi > /dev/null 2>&1 || exit 1 fi # Shared filesystem mounted mountpoint -q /data || exit 1 mountpoint -q /scratch || exit 1 # InfiniBand (if present) if [ -d /sys/class/infiniband ]; then ibstat | grep -q "State: Active" || exit 1 fi exit 0 ``` ### Automatic Node Recovery ```ini # slurm.conf ReturnToService=2 # Nodes rejoin after reboot with any resources ``` --- ## Comparison with Cloud Deployments | Aspect | On-Premise | ParallelCluster | PCS | |--------|-----------|-----------------|-----| | **Hardware control** | Full | EC2 instance types | EC2 instance types | | **Scaling** | Fixed capacity | Auto-scale to zero | Auto-scale to zero | | **Networking** | Custom (IB, OPA) | EFA for HPC | EFA for HPC | | **Storage** | Lustre, BeeGFS, GPFS | FSx Lustre, EFS, EBS | FSx Lustre, EFS | | **Config control** | Full | Partial (deny lists) | Managed | | **HA** | slurmctld backup | Single head node | AWS-managed | | **Maintenance** | You own it all | AWS manages instances | AWS manages Slurm | | **Cost model** | CapEx + power | Pay-per-use | Pay-per-use |

Related Modules

• Installation -- detailed Slurm package installation and auth setup
• Configuration -- slurm.conf deep dive
• High Availability -- slurmctld failover details
• AWS ParallelCluster -- cloud alternative to on-prem
• AWS PCS -- managed Slurm alternative

References

• SchedMD: Quick Start Administrator Guide
• SchedMD: Large Cluster Administration Guide
• SchedMD: Configless Slurm
• SchedMD: slurm.conf Manual
• SchedMD: nss_slurm
• SchedMD: Network Configuration
• SchedMD: Topology Plugin