Why Is It Pending / Why Did It Fail

# Why Is My Job Pending? Why Did It Fail?

Two decision trees. Start at the first command; follow the branch that matches.

---

## Pending: Start Here

```bash
squeue --me -t PD -o '%.10i %.9P %.12j %.2t %.19S %R'   # REASON is the last column
squeue --me --start                                      # expected start time, if backfill is on
sprio -j <jobid>                                         # priority breakdown (multifactor)
```

- `START_TIME` shown (`squeue --start` needs the backfill scheduler)
    - The scheduler has a plan. Nothing to fix; shorten `--time` to fit an earlier gap
- `N/A` or `(null)` start time
    - Read the REASON below
- REASON is empty or `WaitingForScheduling`
    - No reason set yet. Re-run `squeue` in a minute

Only one reason is shown, the one the scheduler hit first. Fix it, and the next may appear.

---

## Pending: Waiting Your Turn

- `Priority`
    - Higher-priority jobs are ahead of yours in the partition or reservation
    - Check `sshare -U` (fairshare) and `sprio -j <jobid>`; a shorter `--time` helps backfill
- `Resources`
    - Your request is valid but the nodes, CPUs, memory or GPUs are in use
    - Compare with `sinfo -o '%P %a %D %T'`; ask for less, or wait
- `BeginTime`
    - You (or a script) set `--begin`. Check `scontrol show job <jobid> | grep StartTime`
- `Dependency`
    - A parent job has not finished: `scontrol show job <jobid> | grep Dependency`
- `DependencyNeverSatisfied`
    - The parent failed. `scancel` and resubmit

---

## Pending: A Limit Is Reached

- `QOSMaxCpuPerUserLimit` (any `QOSMax*`)
    - Your request exceeds a QOS limit (per job, per user). Request less or use a different QOS
- `QOSGrp*` / `AssocGrp*` (for example `AssocGrpCPUMinutesLimit`)
    - The QOS or account has hit an aggregate limit, shared by everyone in it. Wait, or ask the account owner
    - `AssocGrpCPUMinutesLimit` = CPU-minutes budget (past, present and future jobs) is used up
- `AssocMaxJobsLimit`
    - Too many of your jobs running. They start as others finish
- `PartitionTimeLimit`
    - `--time` exceeds the partition maximum. `scontrol update jobid=<id> TimeLimit=<shorter>`, or resubmit
- `Licenses`
    - Waiting for a license. `scontrol show licenses`

`scontrol show job <jobid>` prints the full `Reason=` and the limit values.

---

## Pending: Nodes and Partitions

- `ReqNodeNotAvail`
    - A node the job specifically requires is busy, reserved, `DOWN`, `DRAINED` or not responding
    - `UnavailableNodes:` in the reason names the down ones. Remove `--nodelist` or `--exclude`, or ask an admin
- `PartitionDown` / `PartitionInactive`
    - Partition is not accepting starts. `sinfo -p <partition>`
- `PartitionNodeLimit`
    - Node count outside the partition's current limits
- `Reservation`
    - Waiting for an advanced reservation window
- `JobHeldUser` / `JobHeldAdmin`
    - Held. `scontrol release <jobid>` for your own; ask an admin otherwise
- `InvalidAccount` / `InvalidQOS`
    - Account or QOS is wrong for you: `sacctmgr show assoc user=$USER format=account,qos%30`

> **ParallelCluster Note:** A job on a cloud queue can sit in `CF` (CONFIGURING) for minutes while the node boots. That is not pending; see [Job State Codes](job-state-codes.md).

---

## Failed: Rejected at Submit Time

`sbatch` prints an error and no job id. Nothing was queued.

| You see | Cause | Fix |
|---|---|---|
| Invalid account (or account/partition) | Not in that account, or account missing; some sites require `--account` | `sacctmgr show assoc user=$USER format=account,partition` |
| Invalid QOS specification | QOS name wrong or not allowed for you | List allowed QOS (same command, `qos`) |
| Invalid partition name | Typo, or hidden partition | `sinfo -a` shows hidden partitions |
| Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) | A limit applies, or `--account` missing (LUMI shows this text) | Check the limits for your account and QOS |
| Node count exceeds partition | `--nodes` is larger than the partition holds: rejected, not queued | Reduce `--nodes` |
| Requested time over a limit | `MaxWallDurationPerJob` on the association is denied at submit; a partition limit pends (`PartitionTimeLimit`) | Shorten `--time` |
| Memory or GPU request impossible | More than any node has | `scontrol show node <n> \| grep -E 'RealMemory\|Gres'` |
| Unable to contact slurmctld, socket timed out | Controller busy or down; retry in a minute (NERSC: scheduler busy) | If it persists, ticket |

---

## Failed: Ran, Then Died

```bash
sacct -j <jobid> -o JobID,State,ExitCode,Elapsed,Timelimit,ReqMem,MaxRSS,NodeList%20
```

Read `State` first, `ExitCode` second (`return_code:signal`).

- `TIMEOUT` (`0:0` is normal)
    - Hit `--time`. Raise it, checkpoint, or split the work
- `OUT_OF_MEMORY` / `0:125`
    - The job's memory cgroup limit was hit. Raise `--mem`; see `MaxRSS`
- `137:0` with `FAILED`
    - A program inside the script was killed by signal 9: OOM killer, `scancel --signal`, or a step kill. Check `dmesg`/OOM, `MaxRSS`
- `0:53` with `Reason=RaisedSignal:53(Real-time_signal_19)` (25.x; `JobLaunchFailure` on older releases), `Elapsed` of 0–1 s
    - The batch step never started: the `--output`/`--error` directory (e.g. `logs/`) or work directory is missing on the compute node. `mkdir -p logs` before `sbatch`. No output file exists
- `NODE_FAIL`
    - The node died under your job (`Killing JobId=... on failed node`). Resubmit; check `Restarts=` in `scontrol show job` to see if Slurm already requeued it
- `1:0` / `127:0` / `139:0`
    - Your program: error exit / command not found / segfault. Read the job's stdout and stderr

---

## Before You Open a Ticket

Send these four, in one message:

1. **The script** and the exact submit command (`sbatch ...`)
2. **`module list`** output, from the same shell and script
3. **The job id**, and `sacct -j <jobid> -o JobID,State,ExitCode,Elapsed,NodeList%20`
4. **The error text**, verbatim, plus the time you ran it (`date`)

Add the stdout/stderr files and the output of `scontrol show job <jobid>` (pending jobs) if you have them. This is NERSC's minimum list; it saves one round trip.

Why Is It Pending / Why Did It Fail

References¶