Skip to content

Monitoring Jobs

Exercises

  1. Custom squeue format

Use squeue with a custom output format to show your jobs with these columns: job ID, partition, job name (up to 30 characters), state, elapsed time, time limit, and node list. Submit a test job first so you have something to see.

Hint / Solution 
# Submit a test job
sbatch --time=00:10:00 --wrap="sleep 300" --job-name=format_test

# Custom format
squeue --me -o "%.8i %.9P %.30j %.2t %.10M %.10l %R"
Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0). 
$ squeue --me -o '%.8i %.9P %.30j %.2t %.10M %.10l %R'
   JOBID PARTITION                           NAME ST       TIME TIME_LIMIT NODELIST(REASON)
     679     batch                       val_mg4a CG       0:03       5:00 batch-dy-compute-1
     680     batch                       val_mg4b CG       0:03       5:00 batch-dy-compute-2
     682     batch                 val_mj1_format PD       0:00      10:00 (None)
  1. Inspect a job with scontrol

Submit a job that sleeps for 5 minutes. While it is running, use scontrol show job to find: (a) the working directory, (b) the stdout file path, (c) the exact submit time, and (d) the TRES (trackable resources) allocated.

Hint / Solution 
sbatch --time=00:10:00 --mem=2G --cpus-per-task=2 --wrap="sleep 300"

scontrol show job <jobid>
# Look for: WorkDir, StdOut, SubmitTime, and TRES fields
Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0). 
$ scontrol show job 42
JobId=42 JobName=val_mj2_inspect
   UserId=ec2-user(1000) GroupId=ec2-user(1000) MCS_label=N/A
   Priority=3500 Nice=0 Account=pcdefault QOS=normal
   JobState=CONFIGURING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=00:00:02 TimeLimit=00:10:00 TimeMin=N/A
   SubmitTime=2026-04-15T02:12:24 EligibleTime=2026-04-15T02:12:24
   AccrueTime=2026-04-15T02:12:24
   StartTime=2026-04-15T02:12:24 EndTime=2026-04-15T02:22:24 Deadline=N/A
   SuspendTime=None SecsPreSuspend=0 LastSchedEval=2026-04-15T02:12:24 Scheduler=Main
   Partition=batch AllocNode:Sid=ip-172-31-94-160:199396
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=batch-dy-compute-3
   BatchHost=batch-dy-compute-3
   NumNodes=1 NumCPUs=2 NumTasks=1 CPUs/Task=2 ReqB:S:C:T=0:0:*:*
   ReqTRES=cpu=2,mem=2G,node=1,billing=2
   AllocTRES=cpu=2,mem=2G,node=1,billing=2
   Socks/Node=* NtasksPerN:B:S:C=0:0:*:* CoreSpec=*
   MinCPUsNode=2 MinMemoryNode=2G MinTmpDiskNode=0
   Features=(null) DelayBoot=00:00:00
   OverSubscribe=OK Contiguous=0 Licenses=(null) LicensesAlloc=(null) Network=(null)
   Command=(null)
   SubmitLine=sbatch --time=00:10:00 --mem=2G --cpus-per-task=2 --wrap=sleep 300 --job-name=val_mj2_inspect
   WorkDir=/home/ec2-user/slurm-val-SXAT3v
   StdErr=
   StdIn=/dev/null
   StdOut=/home/ec2-user/slurm-val-SXAT3v/slurm-42.out
   TresPerTask=cpu=2
  1. Find a completed job's memory usage with sacct

After a job completes, use sacct to compare the memory you requested (ReqMem) with the actual peak memory used (MaxRSS). Format the output to include JobID, JobName, ReqMem, MaxRSS, and State.

Hint / Solution 
sacct -j <jobid> --format=JobID,JobName,ReqMem,MaxRSS,State

# Note: MaxRSS appears on the .batch step, not the parent job line
Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0). 
$ sacct -j 42 --format=JobID,JobName,ReqMem,MaxRSS,State
JobID           JobName     ReqMem     MaxRSS      State 
------------ ---------- ---------- ---------- ---------- 
42             val_mj3       512M             COMPLETED 
42.batch         batch                  896K  COMPLETED
  1. Use sinfo to find idle nodes

Use sinfo to find all idle nodes in the default partition. Display hostnames, CPU count, and memory for each idle node.

Hint / Solution 
sinfo -p batch -t idle -o "%n %c %m"
Replace `batch` with your cluster's default partition name if different. Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0). 
$ sinfo -p batch -t idle -o '%n %c %m'
HOSTNAMES CPUS MEMORY
batch-dy-compute-3 8 31129
batch-dy-compute-4 8 31129
batch-dy-compute-1 8 31129
batch-dy-compute-2 8 31129
  1. Track estimated start time of a pending job

Submit a job requesting resources that may cause it to pend (e.g., a large memory request). While it is pending, use squeue --start to check its estimated start time.

Hint / Solution 
sbatch --time=01:00:00 --mem=200G --wrap="sleep 60"

squeue --me --start
# The START_TIME column shows when Slurm estimates the job will begin
Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0). 
$ squeue --me --start
             JOBID PARTITION     NAME     USER ST          START_TIME  NODES SCHEDNODES           NODELIST(REASON)
               685     batch  val_mj5 ec2-user PD                 N/A      1 (null)               (JobHeldUser)

References