Resource Management

# Resource Management

## GRES: Generic Resources

GRES extends Slurm's native CPU/memory management to arbitrary resources -- GPUs, FPGAs, licenses, or custom resources.

### GPU Configuration

In `slurm.conf`:

```ini
GresTypes=gpu
NodeName=gpu[01-08] Gres=gpu:a100:4 CPUs=64 RealMemory=512000
```

> **ParallelCluster Note:** ParallelCluster writes the GPU GRES for you, but not by detection at boot: the GPU count per instance type is **pre-enumerated at cluster create** into an explicit `gres.conf` include chain (`gres.conf` → `slurm_parallelcluster_gres.conf` → per-partition files with `NodeName=… Name=gpu File=/dev/nvidia[0-N]`). Do not add `AutoDetect=nvml`: ParallelCluster nodes are `State=CLOUD`, and Slurm rejects AutoDetect on cloud/dynamic nodes (`slurmctld.log`: `error: Cannot use AutoDetect on cloud/dynamic node`) while `scontrol reconfigure` still exits 0 -- the rejection is silent. The same applies to PCS and CycleCloud. See the [ParallelCluster module](../deployment/aws-parallelcluster.md) for the include chain and what it means for GPU accounting.

> **On-Prem Note:** GPU nodes require manual `gres.conf` configuration (shown below), or `AutoDetect=nvml` if the NVIDIA management library is installed on all GPU nodes **and the nodes are persistent** (not `State=CLOUD`). Nodes registered dynamically with `slurmd -Z` can use AutoDetect; cloud nodes cannot.

In `gres.conf` (on each GPU node, or globally if using AutoDetect):

```ini
# /etc/slurm/gres.conf

# Auto-detect NVIDIA GPUs (persistent/bare-metal nodes)
AutoDetect=nvml

# Explicit configuration (required on State=CLOUD nodes; what ParallelCluster/PCS generate):
# Name=gpu Type=a100 File=/dev/nvidia0 Cores=0-15
# Name=gpu Type=a100 File=/dev/nvidia1 Cores=16-31
# Name=gpu Type=a100 File=/dev/nvidia2 Cores=32-47
# Name=gpu Type=a100 File=/dev/nvidia3 Cores=48-63
```

`AutoDetect=nvml` uses NVIDIA's management library to discover GPUs automatically, including type, cores, and topology. Prefer it on persistent nodes; cloud/dynamic (`State=CLOUD`) nodes must use explicit `File=` entries because the node may be allocated while it is still powered off and nothing can be probed. **26.05+:** `AutoDetect=full` tries every detection plugin (nvml, rsmi, oneapi, …) and AutoDetect also works for nodes created with `slurmd -Z`.

> **Cloud Note (accounting):** because `AutoDetect` cannot run on cloud nodes, the NVML-sourced TRES `gres/gpumem` and `gres/gpuutil` are never populated there. GPU **allocation** accounting (`gres/gpu` in `AllocTRES`) does work once `AccountingStorageTRES` lists it -- see TRES below.

### GPU Isolation with cgroups

Critical for multi-user GPU nodes:

```ini
# cgroup.conf
ConstrainDevices=yes    # Restricts GPU device access per job
```

Without this, a job allocated 1 GPU could access all GPUs on the node. With cgroups, `CUDA_VISIBLE_DEVICES` is set automatically and device access is enforced at the kernel level.

---

## Software License Management

Track scarce software licenses as consumable resources:

```ini
# slurm.conf -- cluster-wide license pool
Licenses=matlab:50,gaussian:20,schrodinger_glide:100
```

Users request licenses at submission:

```bash
sbatch --licenses=matlab:1 my_job.sh
```

Slurm tracks allocations and won't dispatch jobs unless licenses are available.

For external license servers (FLEXlm), use periodic license updates:

```bash
# Update available license count dynamically
scontrol update licenses matlab:45    # Set current available count
```

Or automate with a cron job / scrontab that queries the FLEXlm server.

### Remote licenses

A license configured as a remote resource (`sacctmgr show resource`, `Remote=yes` in `scontrol show licenses`) is synced by a vendor sensor. It binds only to jobs the vendor's job server submits (it sets `--licenses=`). A job submitted with plain `sbatch` shows `(null)` in the licenses column even if the application consumed licenses:

```bash
sacctmgr show resource
scontrol show licenses
squeue -a -h -o '%i %W'
```

Do not infer license demand from that column for direct submissions; use the license server's own usage log.

---

## Memory Management

### Configuration

```ini
# slurm.conf
SelectTypeParameters=CR_Core_Memory    # Track memory as consumable resource
```

```ini
# cgroup.conf
ConstrainRAMSpace=yes       # Enforce memory limits via cgroups
ConstrainSwapSpace=yes      # Include swap in memory accounting
```

### Default Memory per CPU

Set a default so users don't accidentally get the entire node's memory:

```ini
# slurm.conf
DefMemPerCPU=4096    # 4 GB default per CPU (in MB)
```

### Node Memory

Report slightly less than physical RAM to leave room for the OS:

```ini
# For a node with 256 GB physical RAM:
NodeName=node001 RealMemory=250000    # ~244 GB available for jobs
```

---

## Node Features and Constraints

Tag nodes with features for user-driven scheduling:

```ini
# slurm.conf
NodeName=node[001-024] Features=haswell,ethernet,ssd
NodeName=node[025-048] Features=skylake,infiniband
NodeName=gpu[01-04] Features=a100,nvlink,pcie4
NodeName=gpu[05-08] Features=v100,pcie3
```

Users select with `--constraint`:

```bash
sbatch --constraint=infiniband my_mpi_job.sh
sbatch --constraint="a100&nvlink" my_gpu_job.sh
```

### Active Features (Dynamic)

Nodes can change features at boot (e.g., different kernel configurations):

```ini
NodeName=flex[1-10] Features=centos8 ActiveFeatures=centos8
```

---

## Resource Limits Hierarchy

Limits can be set at multiple levels. The most restrictive applies:

```
Partition limits (slurm.conf)
  └── Partition QOS limits (sacctmgr)
        └── Account limits (sacctmgr)
              └── User limits (sacctmgr)
                    └── Job QOS limits (sacctmgr)
```

### Example: Layered Limits

```bash
# Account level: Lab can use 512 CPUs total
sacctmgr modify account smith_lab set GrpTRES=cpu=512

# User level: Each user max 128 CPUs
sacctmgr modify user where account=smith_lab set MaxTRESPerUser=cpu=128

# QOS level: "normal" QOS max 64 CPUs per job
sacctmgr modify qos normal set MaxTRESPerJob=cpu=64
```

Result: A user in smith_lab with qos=normal can submit a job requesting up to 64 CPUs. They can have multiple running jobs up to 128 CPUs total. The lab as a whole can't exceed 512 CPUs.

---

## TRES: Trackable Resources

TRES extends resource accounting beyond CPUs to include memory, GPUs, nodes, licenses, and billing weights. The default `AccountingStorageTRES` is `cpu,mem,energy,node,billing,fs/disk,vmem,pages` -- **GPUs are not tracked until you add them**:

```ini
# slurm.conf -- track GPU allocations (list every typed GRES you use as well)
AccountingStorageTRES=gres/gpu,gres/gpu:a100

# configure billing weights per partition
PartitionName=gpu TRESBillingWeights="CPU=1.0,Mem=0.25G,GRES/gpu=10.0"
```

This means 1 GPU is "worth" 10 CPUs for billing/fairshare purposes. A job using 4 CPUs and 1 GPU is billed as 4 + 10 = 14 billing units. Tracking begins at the `slurmctld` restart that adds the TRES; it is never retroactive, so a GPU partition that has run for a year with the default setting shows only `cpu,mem,node,billing` in `AllocTRES` for every job. Diagnose with `sacctmgr show tres` (is `gres/gpu` listed?) and by comparing `ReqTRES` with `AllocTRES` before you report "GPU usage is zero".

> **ParallelCluster Note:** `AccountingStorageTRES` is **not** on the `CustomSlurmSettings` deny list, so with `SlurmSettings/Database` configured you can add it and get real `gres/gpu` allocation accounting on a stock cluster.

View TRES usage:

```bash
sacct -j 12345 --format=JobID,ReqTRES%40,AllocTRES%60
sacctmgr show tres
sreport cluster utilization
```

---

## cgroup Resource Enforcement

Without cgroups, Slurm resource limits are advisory. With cgroups, they're enforced:

| cgroup Setting | Enforces |
|----------------|----------|
| `ConstrainCores=yes` | CPU core binding |
| `ConstrainRAMSpace=yes` | Memory limit (OOM kill if exceeded) |
| `ConstrainSwapSpace=yes` | Swap + memory combined limit |
| `ConstrainDevices=yes` | GPU/device access isolation |

```ini
# cgroup.conf (recommended production settings)
CgroupPlugin=cgroup/v2
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainSwapSpace=yes
ConstrainDevices=yes
AllowedRAMSpace=100      # % of requested (100 = strict)
AllowedSwapSpace=0       # No swap allowed
```

---

## Recognising a Non-Enforcing Cluster

Some clusters schedule jobs but enforce nothing. The signature is all of these together:

```ini
SelectTypeParameters=CR_CPU
DefMemPerNode=UNLIMITED            # or no DefMemPerCPU/DefMemPerNode
TaskPlugin=task/affinity           # no task/cgroup
ProctrackType=proctrack/linuxproc
JobAcctGatherType=jobacct_gather/none
# cgroup.conf: absent, or ConstrainCores=no / ConstrainRAMSpace=no
```

This is a design state (often a vendor default), not a fault. Describe it as "not enforced".

---

## Non-Enforcing: Consequences

- **Memory is not scheduled.** `CR_CPU` does not make memory consumable; two jobs that each need most of a node's RAM can share it.
- **OOM is node-wide and unattributed.** The kernel picks the victim; Slurm sees SIGKILL and records exit `137`. The victim is not necessarily the job that overran. `OUT_OF_MEMORY` state appears only when Slurm's cgroup plugin sees the kill.
- **`task/affinity` pins but does not confine.** Python `os.cpu_count()` and R `detectCores()` size thread pools from the whole node, so jobs over-thread their pinned cores. Set `OMP_NUM_THREADS=$SLURM_CPUS_PER_TASK`.
- **Node readouts mislead.** `CPULoad` is the OS load average (lagging, not utilisation); `FreeMem` excludes page cache; `AllocMem` is what Slurm allocated, not what was touched.
- **Work outside the scheduler.** Without `pam_slurm_adopt` and `PrologFlags=Contain`, ssh sessions on compute nodes are outside any job. Compare `ps` output against `scontrol listpids`.

---

## Non-Enforcing: Target Configuration

```ini
# slurm.conf
SelectType=select/cons_tres
SelectTypeParameters=CR_Core_Memory
DefMemPerCPU=2048                    # MB per allocated CPU
TaskPlugin=task/cgroup,task/affinity
ProctrackType=proctrack/cgroup
JobAcctGatherType=jobacct_gather/cgroup
PrologFlags=Contain
# nodes: set MemSpecLimit so system processes keep memory

# cgroup.conf
ConstrainCores=yes
ConstrainRAMSpace=yes
ConstrainDevices=yes
ConstrainSwapSpace=yes
```

`ConstrainRAMSpace` only takes effect with a `CR_*_Memory` option. Enabling it kills jobs that silently overran: announce it, set a real `DefMemPerCPU`, and try a test partition first.

> **On-Prem Note:** This is the usual state of vendor-built or inherited clusters.

> **ParallelCluster Note:** ParallelCluster ships cgroup enforcement by default; the signature mostly appears on on-prem clusters.

Exercises¶

Configure a GPU GRES

Add a new GPU node gpu09 with 2 NVIDIA A100 GPUs to the cluster. Define the GRES in both slurm.conf and gres.conf. Use AutoDetect=nvml in gres.conf if gpu09 is a persistent node; show the explicit form you would need if it were a State=CLOUD node.

Hint / Solution

# In slurm.conf, add the node definition:
#   NodeName=gpu09 CPUs=64 RealMemory=512000 Gres=gpu:a100:2

# In gres.conf (on gpu09 or globally):
#   AutoDetect=nvml
# Or explicitly:
#   Name=gpu Type=a100 File=/dev/nvidia0 Cores=0-31
#   Name=gpu Type=a100 File=/dev/nvidia1 Cores=32-63

# Restart slurmctld (new node requires restart)
systemctl restart slurmctld

# Start slurmd on the new node
ssh gpu09 systemctl start slurmd