Monitoring Jobs

# Monitoring Jobs

## squeue: Viewing the Job Queue

`squeue` shows jobs currently in the system (pending, running, completing).

### Your Jobs

```bash
$ squeue --me
JOBID  PARTITION  NAME        USER  ST  TIME   NODES  NODELIST(REASON)
12345  batch      blast_run   jdoe   R  1:23:45    1  node042
12346  batch      analysis    jdoe   R  0:05:12    1  node018
12347  gpu        alphafold   jdoe  PD  0:00       1  (Resources)
```

### All Jobs

```bash
$ squeue
```

### Filter by Partition

```bash
$ squeue -p gpu
```

### Filter by State

```bash
# Only pending jobs
$ squeue -t PENDING

# Only running jobs
$ squeue -t RUNNING
```

### Job State Codes

| Code | State | Meaning |
|------|-------|---------|
| `PD` | PENDING | Waiting for resources |
| `R` | RUNNING | Executing on compute node(s) |
| `CG` | COMPLETING | Job is finishing (cleaning up) |
| `CD` | COMPLETED | Finished successfully (exit code 0) |
| `F` | FAILED | Finished with non-zero exit code |
| `TO` | TIMEOUT | Killed because it exceeded time limit |
| `CA` | CANCELLED | Cancelled by user or admin |
| `OOM` | OUT_OF_MEMORY | Killed for exceeding memory limit |
| `NF` | NODE_FAIL | Node failure during job execution |

### Pending Reasons

When a job shows state `PD`, the reason column explains why:

| Reason | Meaning |
|--------|---------|
| `(Resources)` | Waiting for resources to become available |
| `(Priority)` | Lower priority than other pending jobs |
| `(Dependency)` | Waiting for a dependent job to complete |
| `(QOSMaxJobsPerUserLimit)` | Hit per-user job limit for this QOS |
| `(ReqNodeNotAvail)` | Requested nodes are down or reserved |
| `(PartitionTimeLimit)` | Requested time exceeds partition limit |
| `(AssocMaxJobsLimit)` | Hit account-level job limit |

### Custom Output Formats

```bash
# Compact format showing key fields
$ squeue --me -o "%.8i %.9P %.20j %.8u %.2t %.10M %.6D %R"

# Long format with more detail
$ squeue --me -l

# Show estimated start time for pending jobs
$ squeue --me --start
JOBID  PARTITION  NAME       USER  ST  START_TIME           NODES  NODELIST(REASON)
12347  gpu        alphafold  jdoe  PD  2026-04-12T16:00:00      1  (Resources)
```

Useful format specifiers:

| Specifier | Field |
|-----------|-------|
| `%i` | Job ID |
| `%j` | Job name |
| `%P` | Partition |
| `%u` | Username |
| `%t` | State (compact) |
| `%T` | State (full) |
| `%M` | Time used |
| `%l` | Time limit |
| `%D` | Number of nodes |
| `%R` | Reason (pending) or nodelist (running) |
| `%S` | Start time |

---

## scontrol show job: Detailed Job Information

For full details on a specific job:

```bash
$ scontrol show job 12345
JobId=12345 JobName=blast_run
   UserId=jdoe(1001) GroupId=biolab(2001) MCS_label=N/A
   Priority=4294901742 Nice=0 Account=biolab QOS=normal
   JobState=RUNNING Reason=None Dependency=(null)
   Requeue=1 Restarts=0 BatchFlag=1 Reboot=0 ExitCode=0:0
   RunTime=01:23:45 TimeLimit=04:00:00 TimeMin=N/A
   SubmitTime=2026-04-12T13:00:00 EligibleTime=2026-04-12T13:00:00
   StartTime=2026-04-12T13:06:15 EndTime=2026-04-12T17:06:15
   Partition=batch AllocNode:Sid=login01:54321
   ReqNodeList=(null) ExcNodeList=(null)
   NodeList=node042
   NumNodes=1 NumCPUs=16 NumTasks=1 CPUs/Task=16
   TRES=cpu=16,mem=32G,node=1,billing=16
   MinCPUsNode=16 MinMemoryNode=32G MinTmpDiskNode=0
   Command=/home/jdoe/projects/blast_run.sh
   WorkDir=/home/jdoe/projects
   StdErr=/home/jdoe/projects/logs/blast_run_12345.err
   StdOut=/home/jdoe/projects/logs/blast_run_12345.out
```

This is your go-to command when you need to understand exactly what a job requested and where it's running.

---

## sacct: Completed Job Information

`squeue` only shows active jobs. For completed jobs, use `sacct`:

```bash
# Your jobs from today
$ sacct
JobID       JobName    Partition  Account  AllocCPUS  State      ExitCode
--------    ---------  ---------  -------  ---------  ---------  --------
12340       prep       batch      biolab          4   COMPLETED  0:0
12340.batch batch                 biolab          4   COMPLETED  0:0
12341       analysis   batch      biolab         16   COMPLETED  0:0
12342       failed_run batch      biolab          8   FAILED     1:0
```

### Useful sacct Options

```bash
# Specific job
$ sacct -j 12345

# Jobs from the last 7 days
$ sacct --starttime=now-7days

# Custom format showing resource usage
$ sacct -j 12345 --format=JobID,JobName,Elapsed,MaxRSS,MaxVMSize,State,ExitCode
JobID       JobName    Elapsed    MaxRSS   MaxVMSize  State      ExitCode
--------    ---------  ---------- -------- ---------- ---------  --------
12345       blast_run  01:23:45                       COMPLETED  0:0
12345.batch batch      01:23:45   28456M   32100M     COMPLETED  0:0

# Wide format (no truncation)
$ sacct -j 12345 --format=JobID,JobName%30,Elapsed,MaxRSS,State -l
```

### Key sacct Format Fields

| Field | Description |
|-------|-------------|
| `Elapsed` | Actual wall time used |
| `MaxRSS` | Maximum resident memory (actual usage) |
| `MaxVMSize` | Maximum virtual memory |
| `CPUTime` | CPU time (cores x elapsed) |
| `TotalCPU` | Total CPU time consumed |
| `State` | Final job state |
| `ExitCode` | Exit code (return_code:signal) |
| `ReqMem` | Memory requested |
| `AllocCPUS` | CPUs allocated |
| `NodeList` | Nodes used |

> **Pro tip:** Compare `MaxRSS` to `ReqMem` to see if you're over- or under-requesting memory. This helps you right-size future jobs.

---

## sinfo: Cluster and Partition Status

`sinfo` shows the state of partitions and nodes:

```bash
$ sinfo
PARTITION    AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*          up   7-00:00:00    40  idle  node[001-040]
batch*          up   7-00:00:00     6  mix   node[041-046]
batch*          up   7-00:00:00     2  alloc node[047-048]
gpu             up   2-00:00:00     6  idle  gpu[01-06]
gpu             up   2-00:00:00     2  mix   gpu[07-08]
highmem         up   7-00:00:00     4  idle  bigmem[1-4]
```

### Node States

| State | Meaning |
|-------|---------|
| `idle` | Available, no jobs running |
| `alloc` | Fully allocated to jobs |
| `mix` | Some CPUs allocated, some free |
| `drain` | Marked for maintenance, no new jobs accepted |
| `down` | Unavailable (offline or broken) |
| `idle~` | Powered down, will start on demand (cloud/power-saving) |

### Useful sinfo Variations

```bash
# Summary by partition
$ sinfo -s

# Show node details
$ sinfo -N -l

# Show specific partition
$ sinfo -p gpu

# Show only idle nodes with their resources
$ sinfo -p batch -t idle -o "%n %c %m %G"
HOSTNAMES  CPUS  MEMORY  GRES
node001    64    256000  (null)
node002    64    256000  (null)
```

> **ParallelCluster Note:** You may see nodes in `idle~` state (powered down). These are dynamic nodes that will power on when jobs need them. There may be a few minutes of launch delay for these nodes.

---

## Putting It Together: A Monitoring Workflow

```bash
# 1. Submit your job
$ sbatch analysis.sh
Submitted batch job 12350

# 2. Check it's queued
$ squeue --me
JOBID  PARTITION  NAME      USER  ST  TIME   NODES  NODELIST(REASON)
12350  batch      analysis  jdoe  PD  0:00       1  (Priority)

# 3. Check when it might start
$ squeue --me --start

# 4. Once running, check progress
$ squeue --me
JOBID  PARTITION  NAME      USER  ST  TIME    NODES  NODELIST(REASON)
12350  batch      analysis  jdoe   R  0:12:34     1  node015

# 5. After completion, check results
$ sacct -j 12350 --format=JobID,Elapsed,MaxRSS,State,ExitCode
```

Exercises¶

Custom squeue format

Use squeue with a custom output format to show your jobs with these columns: job ID, partition, job name (up to 30 characters), state, elapsed time, time limit, and node list. Submit a test job first so you have something to see.

Hint / Solution

# Submit a test job
sbatch --time=00:10:00 --wrap="sleep 300" --job-name=format_test

# Custom format
squeue --me -o "%.8i %.9P %.30j %.2t %.10M %.10l %R"

Inspect a job with scontrol

Submit a job that sleeps for 5 minutes. While it is running, use scontrol show job to find: (a) the working directory, (b) the stdout file path, (c) the exact submit time, and (d) the TRES (trackable resources) allocated.

Hint / Solution

sbatch --time=00:10:00 --mem=2G --cpus-per-task=2 --wrap="sleep 300"

scontrol show job <jobid>
# Look for: WorkDir, StdOut, SubmitTime, and TRES fields

Find a completed job's memory usage with sacct

After a job completes, use sacct to compare the memory you requested (ReqMem) with the actual peak memory used (MaxRSS). Format the output to include JobID, JobName, ReqMem, MaxRSS, and State.

Hint / Solution

sacct -j <jobid> --format=JobID,JobName,ReqMem,MaxRSS,State

# Note: MaxRSS appears on the .batch step, not the parent job line

Use sinfo to find idle nodes

Use sinfo to find all idle nodes in the default partition. Display hostnames, CPU count, and memory for each idle node.

Hint / Solution

sinfo -p batch -t idle -o "%n %c %m"

Replace `batch` with your cluster's default partition name if different.

Track estimated start time of a pending job

Submit a job requesting resources that may cause it to pend (e.g., a large memory request). While it is pending, use squeue --start to check its estimated start time.

Hint / Solution

sbatch --time=01:00:00 --mem=200G --wrap="sleep 60"

squeue --me --start
# The START_TIME column shows when Slurm estimates the job will begin

Monitoring Jobs

Exercises¶

References¶