Monitoring & Accounting

<!-- audience: L4 --> <!-- deployment: all --> <!-- sge-source: SGE Admin 05-Monitoring --> # Monitoring & Accounting ## sacct: Job Accounting `sacct` queries the SlurmDBD database for completed job data. ### Common Queries ```bash # Jobs from today sacct # Specific job with resource details sacct -j 12345 --format=JobID,JobName,Partition,Account,User,Elapsed,MaxRSS,MaxVMSize,TotalCPU,AllocCPUS,AllocTRES%60,State,ExitCode # All jobs from a specific account this month sacct --account=smith_lab --starttime=2026-04-01 --format=JobID,User,JobName,Elapsed,State # Failed jobs in the last 7 days sacct --starttime=now-7days --state=FAILED,TIMEOUT,OUT_OF_MEMORY --format=JobID,User,JobName,Elapsed,State,ExitCode # Jobs on a specific partition sacct --partition=gpu --starttime=now-7days --format=JobID,User,Elapsed,AllocTRES%40,State ``` ### Key Format Fields | Field | Description | |-------|-------------| | `Elapsed` | Actual walltime | | `TotalCPU` | CPU time used (user + system) | | `MaxRSS` | Peak resident memory | | `ReqMem` | Requested memory | | `AllocCPUS` | CPUs allocated | | `AllocTRES` | All TRES allocated (CPUs, memory, GPUs) | | `State` | Final state (COMPLETED, FAILED, TIMEOUT, etc.) | | `ExitCode` | Return code and signal | | `CPUTimeRAW` | CPU-seconds (for billing calculations) | --- ## Before you trust sreport Check the accounting data before quoting any `sreport` or `sacct` number. State the result next to every figure you report. **Runaway jobs.** Records slurmdbd thinks are RUNNING but `slurmctld` has forgotten keep accruing time. On clusters we assessed this inflated `sreport` utilization about 2x. Signatures: ```bash sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw,Start,State | wc -l # database view squeue -h -t R | wc -l # controller view grep 'more time than is possible' /var/log/slurm/slurmdbd.log # "We have more time than is possible" ``` Any `sacct` job ID that `squeue` lacks is a runaway candidate. After a database snapshot, an admin runs `sacctmgr show runawayjobs`; it lists them and offers to set an end time (`set EndState=Completed|Failed`). It modifies the database, so never run it unattended. Re-read `sreport` after the fix and quote the post-fix figure. --- ## Before you trust sreport: outages, Down, hidden partitions **Stub rows after an outage.** While slurmdbd was unreachable, slurmctld kept events in memory and flushed minimal records on reconnect: `JobName=allocation`, UID 0, empty Partition and AllocTRES. Counts and wall time survive; user, partition and resource attribution is gone for good. ```bash sacct -a -X -n -P -S <start> -E <end> -o JobIDRaw,JobName,UID,Partition,AllocCPUS,State \ | awk -F'|' '$2=="allocation" || ($3==0 && $4=="")' | wc -l ``` Split analyses into full rows and stub rows; state the stub share beside per-user and per-partition figures. **The Down column.** A single node in `IDLE+NOT_RESPONDING` with no drain reason does not show in `sinfo -R`, yet it can produce a double-digit *Down* share. A *Down* percentage that is the same for 1 and 12 months is a standing node, not an incident. ```bash sinfo -a -R sinfo -a -N -o '%N %T %E' | grep -i -E 'down|drain|not_resp' ``` **Hidden partitions.** `sinfo`, `squeue` and `scontrol show node` omit hidden partitions and ones your group may not use unless you pass `-a`. A census without `-a` under-counts the fleet: `sinfo -a -s`, `scontrol -a show partition`. --- ## sreport: Utilization Reports `sreport` generates summary reports from accounting data. ### Cluster Utilization ```bash # Overall cluster utilization this month sreport cluster utilization start=2026-04-01 # By account sreport cluster AccountUtilizationByUser start=2026-04-01 --tres=cpu # Top users sreport user TopUsage start=2026-04-01 --tres=cpu TopCount=20 ``` ### GPU Utilization > **Prerequisite:** `sreport --tres=gres/gpu` only returns data if `AccountingStorageTRES` in `slurm.conf` lists `gres/gpu` (and every typed GRES you use, e.g. `gres/gpu:a100`). The default value does **not** include GPUs, and tracking starts only from the restart that adds it -- it is never retroactive. On a stock ParallelCluster/PCS cluster the queries below return nothing until you add it (`CustomSlurmSettings` on ParallelCluster). Check with `sacctmgr show tres` and compare `ReqTRES` vs `AllocTRES` in `sacct` before concluding "nobody uses the GPUs". See [Resource Management](resource-management.md). ```bash # GPU utilization by account sreport cluster AccountUtilizationByUser start=2026-04-01 --tres=gres/gpu # Top GPU users sreport user TopUsage start=2026-04-01 --tres=gres/gpu TopCount=10 ``` --- ## Reporting: Job Sizes and Automation ### Job Size Distribution ```bash sreport job SizesByAccount start=2026-04-01 PrintJobCount ``` ### Scheduled Reports Combine with `scrontab` for automated reporting: ``` #SCRON --time=00:10:00 #SCRON --mem=1G #SCRON -J weekly_report 0 8 * * 1 /opt/slurm/scripts/generate_weekly_report.sh ``` Two rules for any report pulled from `sacct`: - **One month per query, paced.** A large pull loads slurmdbd and the database the live scheduler depends on. Loop over months and sleep at least twice the previous chunk's duration (minimum 5 s); if a chunk nears about 90 s, split it by week. - **Dedupe by `JobIDRaw` at month boundaries.** A job spanning a boundary appears in both monthly pulls. Assert uniqueness before summing: ```bash cut -d'|' -f1 all-months.psv | sort | uniq -d | head # must print nothing ``` --- ## sdiag: Scheduler Diagnostics `sdiag` provides statistics about the scheduler's performance: ```bash $ sdiag ******************************************************* sdiag output at Mon Apr 12 14:30:00 2026 Data since Mon Apr 05 00:00:00 2026 ******************************************************* Server thread count: 5 Agent queue size: 0 Jobs submitted: 12345 Jobs started: 11890 Jobs completed: 11500 Jobs failed: 340 Jobs cancelled: 50 Main schedule statistics (microseconds): Last cycle: 45123 Mean cycle: 38456 Max cycle: 234567 Total cycles: 8640 Backfilling stats Total backfilled jobs (since last stats cycle): 3456 Total backfilled heterogeneous job components: 0 Total cycle count: 8640 Last cycle when: Mon Apr 12 14:29:45 2026 Last cycle: 12345 Mean cycle: 10234 Last depth cycle: 1500 Last depth cycle (try sched): 200 Last queue length: 456 ``` Key metrics to watch: - **Mean cycle time:** How long each scheduling pass takes (should be <1 second for most clusters) - **Backfilled jobs:** Higher is better -- indicates the backfill scheduler is working - **Agent queue size:** Should be 0 or near 0; high values indicate communication delays - **`sdiag` counters reset at midnight UTC** (and on `sdiag --reset`): record the reset time (`Data since`) with every reading before comparing two of them. --- ## Log Files | Log | Location | Content | |-----|----------|---------| | slurmctld | `/var/log/slurm/slurmctld.log` | Controller events, scheduling decisions, errors | | slurmd | `/var/log/slurm/slurmd.log` | Per-node job launch, cgroup, process tracking | | slurmdbd | `/var/log/slurm/slurmdbd.log` | Database operations, accounting | ### Adjusting Log Level ```bash # Temporarily increase verbosity (no restart needed) scontrol setdebug debug2 # Reset to normal scontrol setdebug info # Or set in slurm.conf permanently SlurmctldDebug=info ``` --- ## Log Rotation Add to `/etc/logrotate.d/slurm` **on every node** (controller, database host and compute nodes all write logs). This is the sample from the LOGGING section of `slurm.conf(5)`: ``` /var/log/slurm/*.log { compress missingok nocopytruncate nodelaycompress nomail notifempty noolddir rotate 5 sharedscripts size=5M create 640 slurm root postrotate pkill -x --signal SIGUSR2 slurmctld pkill -x --signal SIGUSR2 slurmd pkill -x --signal SIGUSR2 slurmdbd exit 0 endscript } ``` > **Why SIGUSR2, not SIGHUP:** per `slurmctld(8)` and `slurmdbd(8)`, `SIGUSR2` "rereads the log level from the configs, and then reopens the log file. This should be used when setting up logrotate". `SIGHUP` reloads the configuration (like `scontrol reconfigure`) and does **not** reopen the log. A `postrotate` that sends `SIGHUP` leaves the daemon writing into the renamed `slurmctld.log.1` -- or into a deleted inode that still consumes disk (`lsof +L1 | grep slurm`). After installing the file, force one rotation (`logrotate -f /etc/logrotate.d/slurm`) and confirm the new `slurmctld.log` is the one growing. > **Unrotated logs fill disks.** With no `/etc/logrotate.d/slurm*`, `slurmctld.log` (and the scheduler log if `SlurmSchedLogLevel=1`) can grow by tens to hundreds of MB per day during an error flood. Check, and alert on free space of both filesystems: > > ```bash > ls -la /etc/logrotate.d/ | grep -i slurm > du -sh /var/log/slurm; df -h /var/log /var/spool > ``` > > A full `StateSaveLocation` filesystem is worse than a lost log: the controller can stop saving state. Treat `slurmdbd.log` the same way. --- ## External Monitoring Integration > **ParallelCluster Note:** ParallelCluster publishes head node and compute fleet metrics to CloudWatch automatically (node counts, job states, scaling events). Use CloudWatch alongside Slurm-native tools like `sdiag` and `sreport` for a complete picture. CloudWatch Alarms can notify on unhealthy nodes or scaling failures. > **PCS Note:** PCS integrates with CloudWatch for control-plane and compute metrics out of the box. Since you don't have access to slurmctld logs directly, CloudWatch is the primary tool for monitoring cluster health and scaling behavior. ### Prometheus + Grafana The `slurm-exporter` (community project) exposes Slurm metrics for Prometheus scraping: - Node state counts (idle, allocated, drained, down) - Partition utilization - Job queue depth - Scheduler performance - Fairshare data ### Job Completion Plugins Slurm can push job completion data to external systems: ```ini # slurm.conf JobCompType=jobcomp/elasticsearch JobCompLoc=http://elasticsearch:9200 ``` Or use file-based logging: ```ini JobCompType=jobcomp/filetxt JobCompLoc=/var/log/slurm/job_completions ``` #### When there is no slurmdbd Without accounting, job history and utilisation can still be rebuilt from `slurmctld.log`, a `jobcomp/filetxt` file and `sdiag`/`scontrol` counters. Every figure is a floor or a ceiling, never a measurement; label it. Line shapes below were observed on Slurm 22.05 -- sample real lines (`grep -m5 -E 'Allocate|_start_job' slurmctld.log`) before writing a parser. **Job starts** come in two shapes: ``` sched: Allocate JobId=<id> NodeList=<nodes> #CPUs=<N> Partition=<P> sched/backfill: _start_job: Started JobId=<id> in <partition> on <nodes> ``` The backfill form carries **no CPU count**, so core-hours from the log cover only jobs started by the main scheduler (or must be estimated, and labelled as such). --- **Job ends:** ``` _job_complete: JobId=<id> WEXITSTATUS <n> _job_complete: JobId=<id> WTERMSIG <n> _job_complete: JobId=<id> done Time limit exhausted for JobId=<id> ``` Array tasks log as `JobId=<base>_<idx>(<jobid>)`; key on the numeric id in parentheses. **Submission counter:** record `NEXT_JOB_ID` with the date on every visit. ```bash scontrol show config | grep -i NEXT_JOB_ID ``` The delta between two dated values, divided by the days elapsed, estimates jobs submitted per day (it counts every allocated ID, array tasks included). --- **`jobcomp/filetxt`** writes one `key=value` line per finished job (`JobId= UserId= Name= JobState= Partition= TimeLimit= Start= End= NodeList= ...`) to `JobCompLoc`. Check what is actually configured: ```bash scontrol show config | grep -Ei 'JobComp|AccountingStorage' ``` > **On-Prem Note:** the file is typically created world-readable and holds per-user rows. Set restrictive permissions on `JobCompLoc`, or report a world-readable one as a data-exposure finding. `JobCompType=jobcomp/none` writes nothing. --- ## Accounting Best Practices 1. **Enable accounting from day one.** Retroactive accounting is impossible. 2. **Archive, then purge, old data.** `Purge*After` alone **deletes** records permanently; the oldest record in `sacct` becomes your purge horizon. Configure archiving first: ```ini # slurmdbd.conf ArchiveDir=/var/spool/slurm/archive ArchiveJobs=yes ArchiveSteps=yes ArchiveEvents=yes PurgeEventAfter=12months PurgeJobAfter=24months PurgeStepAfter=6months ``` Verify what a cluster actually does with `sacctmgr show config | grep -Ei 'purge|archive'`, and snapshot the database before a month boundary if you are about to change these values. 3. **Use `sreport` for management** rather than raw `sacct` queries. It's designed for summary reporting. 4. **Monitor scheduler health** with `sdiag` regularly. Increasing cycle times indicate scaling issues. 5. **Set up automated reports** for stakeholders who need utilization data for budgets and planning.

Exercises

1. Generate a cluster utilization report

Use sreport to produce an overall cluster utilization report for the current month. Identify the percentage of CPU time that was allocated vs. idle.

Hint / Solution

# Overall cluster utilization
sreport cluster utilization start=2026-04-01

# Output shows columns: Allocated, Down, PLND Down (planned), Idle, Reserved, Reported
# Allocated / Reported = utilization percentage

# Break down by partition
sreport cluster utilization start=2026-04-01 -t percent

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sreport cluster utilization start=now-7days
--------------------------------------------------------------------------------
Cluster Utilization 2026-10-12T01:00:00 - 2026-10-19T00:59:59
Usage reported in CPU Minutes
--------------------------------------------------------------------------------
  Cluster Allocate     Down PLND Dow     Idle  Planned Reported 
--------- -------- -------- -------- -------- -------- -------- 
slurm-tr+       14        0   282356   201437       34   483840 

1. Find the top 5 users by CPU hours

Generate a report showing the five heaviest CPU consumers this month. Include both their CPU hours and the account they charged to.

Hint / Solution

sreport user TopUsage start=2026-04-01 --tres=cpu TopCount=5

# For GPU hours instead:
sreport user TopUsage start=2026-04-01 --tres=gres/gpu TopCount=5

# For a specific account's users only:
sreport user TopUsage start=2026-04-01 --tres=cpu TopCount=5 Accounts=smith_lab

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sreport user TopUsage start=now-7days --tres=cpu TopCount=5
--------------------------------------------------------------------------------
Top 5 Users 2026-10-12T01:00:00 - 2026-10-19T00:59:59 (604800 secs)
Usage reported in TRES Minutes
--------------------------------------------------------------------------------
  Cluster     Login     Proper Name         Account      TRES Name     Used 
--------- --------- --------------- --------------- -------------- -------- 
slurm-tr+  ec2-user AWS ParallelCl+       pcdefault            cpu       13 
slurm-tr+ testuser1                        research            cpu        0 

1. Check scheduler health with sdiag

Run sdiag and assess the scheduler's performance. Determine the mean scheduling cycle time, how many jobs have been backfilled, and whether the agent queue is healthy.

Hint / Solution

sdiag

# Key things to check:
# 1. "Mean cycle" under "Main schedule statistics" -- should be well under 1,000,000 us (1 second)
# 2. "Total backfilled jobs" -- a healthy number indicates backfill is working
# 3. "Agent queue size" -- should be 0 or near 0; high values mean communication delays
# 4. "Last cycle" timestamp -- should be very recent

# Reset stats to track from a known point:
sdiag --reset

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sdiag
*******************************************************
sdiag output at Mon Oct 19 01:05:33 2026 (1792371933)
Data since      Mon Oct 19 01:05:29 2026 (1792371929)
*******************************************************
Server thread count:  1
RPC queue enabled:    0
Agent queue size:     0
Agent count:          0
Agent thread count:   0
DBD Agent queue size: 0

Jobs submitted: 0
Jobs started:   0
Jobs completed: 0
Jobs canceled:  0
Jobs failed:    0

Job states ts:  Mon Oct 19 01:05:29 2026 (1792371929)
Jobs pending:   0
Jobs running:   0

Main schedule statistics (microseconds):
 Last cycle:   17
 Max cycle:    17
 Total cycles: 1
 Mean cycle:   17
 Mean depth cycle:  0
 Last queue length: 0

Main scheduler exit:
 End of job queue: 1
 Hit default_queue_depth: 0
 Hit sched_max_job_start: 0
 Blocked on licenses: 0
 Hit max_rpc_cnt: 0
 Timeout (max_sched_time): 0

Backfilling stats
 Total backfilled jobs (since last slurm start): 0
 Total backfilled jobs (since last stats cycle start): 0
 Total backfilled heterogeneous job components: 0
 Total cycles: 0
 Last cycle when: Mon Oct 19 00:51:48 2026 (1792371108)
 Last cycle: 0
 Max cycle:  0
 Last depth cycle: 0
 Last depth cycle (try sched): 0
 Last queue length: 0
 Last table size: 0

Backfill exit
 End of job queue: 0
 Hit bf_max_job_start: 0
 Hit bf_max_job_test: 0
 System state changed: 0
 Hit table size limit (bf_node_space_size): 0
 Timeout (bf_max_time): 0

Latency for 1000 calls to gettimeofday(): 27 microseconds

Remote Procedure Call statistics by message type
 REQUEST_PARTITION_INFO                  ( 2009) count:6      ave_time:58     total_time:352
 REQUEST_NODE_INFO                       ( 2007) count:6      ave_time:158    total_time:949
 REQUEST_UPDATE_NODE                     ( 3002) count:5      ave_time:147    total_time:738
 REQUEST_PING                            ( 1008) count:3      ave_time:60     total_time:182
 REQUEST_BUILD_INFO                      ( 2001) count:2      ave_time:151    total_time:303

Remote Procedure Call statistics by user
 ec2-user        (    1000) count:17     ave_time:105    total_time:1786
 root            (       0) count:5      ave_time:147    total_time:738

Pending RPC statistics
 No pending RPCs

1. Find jobs that exceeded their memory request

Use sacct to find completed jobs from the past 7 days where the actual peak memory usage (MaxRSS) was close to or exceeded the requested memory. Identify jobs that were killed by OOM.

Hint / Solution

# Find jobs killed by OOM directly
sacct --starttime=now-7days --state=OUT_OF_MEMORY \
    --format=JobID,User,JobName,ReqMem,MaxRSS,Elapsed,State,ExitCode

# Find completed jobs where MaxRSS was high relative to request
# (look for jobs where MaxRSS approaches ReqMem)
sacct --starttime=now-7days --state=COMPLETED \
    --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed \
    | sort -k5 -h | tail -20

# Find jobs killed by signal 9 (SIGKILL -- often OOM even if not tagged)
sacct --starttime=now-7days \
    --format=JobID,User,JobName,State,ExitCode | grep "0:9"

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sacct --starttime=now-7days --state=OUT_OF_MEMORY --format=JobID,User,JobName,ReqMem,MaxRSS,Elapsed,State,ExitCode
JobID             User    JobName     ReqMem     MaxRSS    Elapsed      State ExitCode 
------------ --------- ---------- ---------- ---------- ---------- ---------- -------- 

$ sacct --starttime=now-7days --state=COMPLETED --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed | head -10
JobID        User       JobName                  ReqMem     MaxRSS    Elapsed 
------------ ---------- -------------------- ---------- ---------- ---------- 

1. Force a log rotation and prove the new file grows

Use the SIGUSR2 postrotate from the Log Rotation section. Force one rotation, then show that the daemon writes to the new slurmctld.log and that no deleted log is still held open. Needs root on the controller (a harness runs this only when root: skip_unless_root).

Hint / Solution

sudo ls -la /var/log/slurm/slurmctld.log*          # note size/inode of the live file
sudo logrotate -f /etc/logrotate.d/slurm           # force one rotation
scontrol ping >/dev/null; sbatch --wrap="true"     # generate log traffic
sudo ls -li /var/log/slurm/slurmctld.log*          # new log: new inode, size growing; .1 is not
sleep 5; sudo ls -la /var/log/slurm/slurmctld.log* # run twice: only the new file should grow
sudo lsof +L1 | grep slurm                         # expect no output (no deleted-but-open logs)

If `slurmctld.log.1` keeps growing or `lsof +L1` lists a slurm log, the `postrotate` is not sending `SIGUSR2` to that daemon.

1. Compare accounting's running jobs with the controller's

Count the jobs slurmdbd believes are running and the jobs slurmctld has running. Decide whether the difference is a runaway-job problem.

Hint / Solution

sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw,Start,State | wc -l   # database view
squeue -h -t R | wc -l                                                      # controller view

# Which IDs does the database hold that the controller does not?
comm -23 <(sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw | sort) \
         <(squeue -h -t R -o %i | sort)

Equal counts (allow a one-second race) mean no runaways. Any ID listed by `comm` is a runaway candidate; the fix (`sacctmgr show runawayjobs`, after a database snapshot) is an admin action -- do not run it in this exercise. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw,Start,State | wc -l
0

$ squeue -h -t R | wc -l
0

$ comm -23 <(sacct -a -X -n -P -s RUNNING -S 2000-01-01 -o JobIDRaw | sort) <(squeue -h -t R -o %i | sort)

1. Trace one job's lifecycle in slurmctld.log

Submit a short job and follow its JobId through slurmctld.log from submission to completion. Name the start-line shape you find (main scheduler or backfill) and whether it carries a CPU count.

Hint / Solution

LOG=$(scontrol show config | awk '/SlurmctldLogFile/ {print $3}')
JOB=$(sbatch --parsable --wrap="sleep 5")
sleep 15
grep -E "JobId=${JOB}( |\$)|JobId=${JOB}_" "$LOG"

# Shapes to look for (sample real lines first; they vary by version):
#   sched: Allocate JobId=<id> NodeList=... #CPUs=N       <- main scheduler, has CPU count
#   sched/backfill: _start_job: Started JobId=<id> in ... <- backfill, NO CPU count
#   _job_complete: JobId=<id> WEXITSTATUS 0
#   _job_complete: JobId=<id> done

The log must be readable by your user and `SlurmctldDebug` at `info` or higher; otherwise run as root. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sudo grep -E 'JobId=42( |$)|JobId=42_' /var/log/slurmctld.log
[2026-10-19T01:05:34.308] _slurm_rpc_submit_batch_job: JobId=42 SLUID=sBHSM58V2H3400 InitPrio=3500 usec=490
[2026-10-19T01:05:35.007] sched: Allocate JobId=42 NodeList=batch-dy-compute-1 #CPUs=1 Partition=batch
[2026-10-19T01:06:01.027] POWER: no more nodes to resume for job JobId=42
[2026-10-19T01:08:29.007] job_time_limit: Configuration for JobId=42 complete
[2026-10-19T01:08:29.007] Resetting JobId=42 start time for node power up
[2026-10-19T01:08:34.358] _job_complete: JobId=42 WEXITSTATUS 0
[2026-10-19T01:08:34.358] _job_complete: JobId=42 done

References

• SchedMD: sacct man page
• SchedMD: sreport man page
• SchedMD: sdiag man page
• SchedMD: Accounting and Resource Limits
• SchedMD: Elasticsearch Guide
• SchedMD: Metrics Guide