Skip to content

Troubleshooting

Exercises

  1. Diagnose why a job is pending

Submit a job to the gpu partition requesting 8 GPUs (more than any single node has if nodes have 4 each) and observe the pending reason. Use squeue to identify the reason code, then determine how to fix the request.

Hint / Solution 
# Submit a job requesting more GPUs than a single node has
sbatch -p gpu --gres=gpu:8 --nodes=1 --wrap="sleep 120"

# Check the pending reason
squeue -u $USER -o "%.8i %.9P %.20j %.2t %.30R"
# Expected reason: (ReqNodeNotAvail) or (Resources)

# Get more details
scontrol show job <jobid> | grep Reason

# Fix: either request fewer GPUs per node, or span multiple nodes:
#   --gres=gpu:4 --nodes=2 --ntasks-per-node=1
scancel <jobid>
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ squeue -j 42 -o '%.8i %.9P %.20j %.2t %.30R'
   JOBID PARTITION                 NAME ST               NODELIST(REASON)
     42     batch              val_ts1 PD                  (JobHeldUser)

$ scontrol show job 42 | grep -E '(JobState|Reason)'
   JobState=PENDING Reason=JobHeldUser Dependency=(null)
  1. Drain a node and verify it stops accepting jobs

Drain node cpu010 with a reason string. Confirm the node shows as draining/drained in sinfo. Submit a job targeting that specific node and verify it won't run there.

Hint / Solution 
# Drain the node
scontrol update NodeName=cpu010 State=DRAIN Reason="exercise: testing drain"

# Verify the state
sinfo -n cpu010
# State should show drain, drng (draining), or drained

# Check the reason
scontrol show node cpu010 | grep -i "state\|reason"

# Try to submit a job to that node -- it should pend
sbatch --nodelist=cpu010 --wrap="hostname"
squeue -u $USER
# Job will pend with reason (ReqNodeNotAvail) until the node is resumed

# Clean up
scancel -u $USER
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-2 State=DRAIN Reason='val: ts2 drain test'

$ sinfo -n debug-dy-small-2
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      0    n/a 
debug        up   infinite      1 drain~ debug-dy-small-2
gpu          up   infinite      0    n/a 
highmem      up   infinite      0    n/a 

$ scontrol show node debug-dy-small-2 | grep -E '(State|Reason)'
   State=IDLE+CLOUD+DRAIN+POWERED_DOWN ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
   Reason=val: ts2 drain test [root@2026-10-19T01:14:47]
  1. Bring a drained node back online

After the previous exercise, resume node cpu010 and verify it returns to an idle state and can accept new jobs.

Hint / Solution 
# Resume the node
scontrol update NodeName=cpu010 State=RESUME

# Verify it's back to idle
sinfo -n cpu010
# State should show idle (or idle* if no slurmd running in a lab environment)

scontrol show node cpu010 | grep -i state
# State=IDLE

# Test that it accepts jobs
srun --nodelist=cpu010 hostname
# Should return: cpu010
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sudo /opt/slurm/bin/scontrol update NodeName=debug-dy-small-2 State=RESUME

$ scontrol show node debug-dy-small-2 | grep State=
   State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A
  1. Find a job that was killed by OOM

Search completed jobs from the past 7 days for any that ended in the OUT_OF_MEMORY state. For each, determine how much memory was requested vs. how much was actually used at peak. Then find jobs whose program died of SIGKILL inside a script that itself exited normally (ExitCode=137:0) -- these are OOM or scancel --signal victims that do not carry the OUT_OF_MEMORY state.

Hint / Solution 
# Find OOM-killed jobs (cgroup OOM: State=OUT_OF_MEMORY, ExitCode=0:125)
sacct --starttime=now-7days --state=OUT_OF_MEMORY \
    --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed,ExitCode,NodeList

# For a specific job, get the full picture
sacct -j <jobid> --format=JobID,JobName,State,ExitCode,ReqMem,MaxRSS,MaxVMSize,AllocCPUS

# MaxRSS close to or at ReqMem confirms memory exhaustion

# Programs killed by SIGKILL inside an otherwise-normal script: 137:0, State=FAILED
sacct --starttime=now-7days -X --format=JobID,User,State,ExitCode | grep '137:0'
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sacct --starttime=now-7days --state=OUT_OF_MEMORY --format=JobID%-12,User%-10,JobName%-20,ReqMem,MaxRSS,Elapsed,ExitCode,NodeList
JobID        User       JobName                  ReqMem     MaxRSS    Elapsed ExitCode        NodeList 
------------ ---------- -------------------- ---------- ---------- ---------- -------- --------------- 

$ sacct --starttime=now-7days --format=JobID,User,State,ExitCode | grep '0:9' | head -10

$ sacct --starttime=now-7days --format=JobID,User,State,ExitCode | grep '0:9' | head -10 || echo '(no SIGKILL exits in window)'
  1. Investigate a node in DOWN state

Using scontrol, examine a node that shows as down or down*. Determine the reason it went down, check whether slurmd is running on it, and bring it back if possible.

Hint / Solution 
# Find down nodes
sinfo -N -l | grep down

# Get details on a specific down node
scontrol show node cpu005 | grep -i "state\|reason\|lastbusy"

# Check if slurmd is running on the node
ssh cpu005 systemctl status slurmd

# If slurmd is stopped, restart it
ssh cpu005 systemctl restart slurmd

# If slurmd is running but the node is still down*, clear the state
scontrol update NodeName=cpu005 State=RESUME

# Verify recovery
sinfo -n cpu005
Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sinfo -N -l | head -20
Mon Oct 19 01:14:51 2026
NODELIST            NODES PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
batch-dy-compute-1      1    batch*        idle 8       8:1:1  31129        0   1000 dynamic, none                
batch-dy-compute-2      1    batch*       idle~ 8       8:1:1  31129        0   1000 dynamic, none                
batch-dy-compute-3      1    batch*       idle~ 8       8:1:1  31129        0   1000 dynamic, none                
batch-dy-compute-4      1    batch*       idle~ 8       8:1:1  31129        0   1000 dynamic, none                
debug-dy-small-1        1     debug       idle~ 2       2:1:1   3891        0   1000 dynamic, none                
debug-dy-small-2        1     debug       idle~ 2       2:1:1   3891        0   1000 dynamic, none                
gpu-dy-t4-1             1       gpu       idle~ 4       4:1:1  15564        0   1000 dynamic, none                
highmem-dy-large-1      1   highmem       idle~ 8       8:1:1  31129        0   1000 dynamic, none                

$ sinfo -R --noheader || echo '(no nodes with REASON set)'
  1. Reproduce 0:53 deterministically

A missing --output directory makes the batch step fail at launch. Reproduce it, then read the record.

Hint / Solution 
JOB=$(sbatch --parsable -o /nonexistent/%j.out --wrap=hostname)
sleep 15
scontrol show job $JOB | grep -E 'JobState|Reason|ExitCode|RunTime'
# Expect: JobState=FAILED, Reason=RaisedSignal:53(Real-time_signal_19) (25.x) or JobLaunchFailure (older), ExitCode=0:53, RunTime of 0-1 s
# sacct needs slurmdbd; without it, scontrol (above) and the slurmctld log are the record
sacct -j $JOB -X -o JobID,State,ExitCode,Reason
Do not read `53` as a signal: the batch step never started. Fix by creating the directory (or use `--chdir=/tmp` when submitting from a directory that does not exist on the node, such as an SSM session cwd). Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ scontrol show job 113 | grep -E 'JobState|Reason|ExitCode|RunTime'
   JobState=FAILED Reason=RaisedSignal:53(Real-time_signal_19) Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:53
   RunTime=00:00:00 TimeLimit=UNLIMITED TimeMin=N/A
  1. Fingerprint the slurmctld log

Rank error signatures by count. Read the top five; one or two usually dominate.

Hint / Solution 
# path differs on ParallelCluster: /var/log/slurmctld.log
cd /var/log/slurm 2>/dev/null || cd /var/log
sudo grep -hE 'error|fatal' slurmctld.log* | sed -E 's/[0-9]+/N/g' | sort | uniq -c | sort -rn | head -20
# compressed rotations: use zgrep -hE instead of grep -hE
Match each top line against the signature table in Daemon Log Hygiene. Counts are lines, not events. An empty result on a healthy cluster is a valid outcome. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ cd /var/log 2>/dev/null; sudo zgrep -hE 'error|fatal' slurmctld.log* 2>/dev/null | sed -E 's/[0-9]+/N/g' | sort | uniq -c | sort -rn | head -20
     45 [N-N-NTN:N:N.N] error: chdir(/var/log): Permission denied
     45 [N-N-NTN:N:N.N] error: Configured MailProg is invalid
      8 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node batch-dy-compute-N
      4 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node debug-dy-small-N
      4 [N-N-NTN:N:N.N] error: _find_node_record: lookup failure for node "val_nodeN"
      4 [N-N-NTN:N:N.N] error: Node val_nodeN has vanished from configuration
      2 [N-N-NTN:N:N.N] slurmscriptd: error: run_command: MailProg can not be executed (/bin/mail) No such file or directory
      2 [N-N-NTN:N:N.N] slurmscriptd: error: _run_script: JobId=N MailProg exit status N:N
      2 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node highmem-dy-large-N
      2 [N-N-NTN:N:N.N] error: gres_node_state_unpack: no plugin configured to unpack data type N from node gpu-dy-tN-N
      2 [N-N-NTN:N:N.N] error: MailProg returned error, it's output was 'Run command failed - configuration error'
      1 [N-N-NTN:N:N.N] error: Nodes batch-dy-compute-[N-N] not responding
  1. Elapsed-bucket triage over the last 7 days

Bucket non-successful jobs by runtime to separate launch/config failures from wasted allocations. Needs slurmdbd.

Hint / Solution 
sacct -a -X -S now-7days -o JobID,State,ExitCode,ElapsedRaw -P -n \
| awk -F'|' '$2!~/COMPLETED|RUNNING|PENDING/ {
    b = ($4<60)?"<1m":($4<600)?"1-10m":($4<3600)?"10-60m":">1h";
    c[$3" "b]++ }
  END { for (k in c) print c[k], k }' | sort -rn | head -20
Run exercise 6 first and re-run this: the `0:53` job appears in the `<1m` bucket. Empty output means no failures in the window. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1). 
$ sacct -a -X -S now-7days -o JobID,State,ExitCode,ElapsedRaw -P -n | awk -F'|' '$2!~/COMPLETED|RUNNING|PENDING/ { b = ($4<60)?"<1m":($4<600)?"1-10m":($4<3600)?"10-60m":">1h"; c[$3" "b]++ } END { for (k in c) print c[k], k }' | sort -rn | head -20
20 0:0 <1m
8 0:15 <1m
3 1:0 <1m
2 0:53 <1m
2 0:125 <1m

References