Managing Jobs

# Managing Jobs

## scancel: Canceling Jobs

### Cancel a Specific Job

```bash
$ scancel 12345
```

### Cancel All Your Jobs

```bash
$ scancel --me
```

### Cancel by Name

```bash
$ scancel --name=blast_run
```

### Cancel by State

```bash
# Cancel all your pending jobs (keep running ones)
$ scancel --me --state=PENDING

# Cancel all your jobs in a specific partition
$ scancel --me --partition=gpu
```

### Cancel Specific Array Tasks

```bash
# Cancel task 5 of array job 12340
$ scancel 12340_5

# Cancel tasks 1-10
$ scancel 12340_[1-10]
```

### Send a Signal Instead of Killing

By default, `scancel` sends `SIGKILL`. To give your job a chance to clean up:

```bash
# Send SIGTERM first (allows graceful shutdown)
$ scancel --signal=TERM 12345

# After a delay, follow up with kill if needed
$ scancel 12345
```

---

## scontrol hold / release: Pausing Job Dispatch

You can prevent a pending job from being scheduled without canceling it:

### Hold a Job

```bash
$ scontrol hold 12345
```

The job stays in the queue but won't be dispatched until released. State shows as `PD` with reason `(JobHeldUser)`.

### Release a Held Job

```bash
$ scontrol release 12345
```

The job returns to normal pending status and will be scheduled when resources are available.

> **Use case:** You submitted a job but realized you need to fix something in the input files. Hold the job, fix the files, then release it -- no need to cancel and resubmit.

---

## scontrol update: Modifying Pending Jobs

You can modify certain attributes of a **pending** job (not running jobs):

### Change Time Limit

```bash
$ scontrol update JobId=12345 TimeLimit=8:00:00
```

### Change Partition

```bash
$ scontrol update JobId=12345 Partition=highmem
```

### Change Job Name

```bash
$ scontrol update JobId=12345 JobName=new_name
```

### Change Account

```bash
$ scontrol update JobId=12345 Account=other_lab
```

### Change Resource Requests

```bash
# Change number of CPUs
$ scontrol update JobId=12345 NumCPUs=32

# Change memory
$ scontrol update JobId=12345 MinMemoryNode=64G
```

> **Limitation:** Once a job is running, most attributes cannot be changed. The time limit is an exception -- administrators can extend it for running jobs.

---

## scontrol requeue: Restarting Jobs

If a job needs to be rerun (e.g., it was affected by a node failure):

```bash
$ scontrol requeue 12345
```

This cancels the running job and puts it back in the pending queue. The job must have `--requeue` enabled (which is the default on most clusters).

---

## Practical Scenarios

### "I submitted the wrong script"

```bash
# Cancel and resubmit
$ scancel 12345
$ sbatch corrected_script.sh
```

### "I need more time"

```bash
# For a pending job, update directly
$ scontrol update JobId=12345 TimeLimit=24:00:00

# For a running job, ask your admin (users typically can't extend running jobs)
```

### "I want to submit jobs but not start them yet"

```bash
# Submit in held state
$ sbatch --hold my_job.sh
Submitted batch job 12346

# Later, release when ready
$ scontrol release 12346
```

### "I need to cancel everything and start over"

```bash
# Cancel all your jobs
$ scancel --me

# Verify nothing is left
$ squeue --me
# (should show no jobs)
```

### "My job is pending forever"

```bash
# Check why
$ squeue --me -o "%.8i %.9P %.20j %.2t %R"
JOBID  PARTITION  NAME                ST  REASON
12347  gpu        alphafold            PD  (Resources)

# Get more details
$ scontrol show job 12347 | grep -i reason
   Reason=Resources

# Check partition availability
$ sinfo -p gpu
PARTITION  AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpu           up  2-00:00:00     8  alloc gpu[01-08]

# All GPU nodes are in use -- you need to wait
```

---

## Job Lifecycle Summary

---

## Command Quick Reference

| Action | Command |
|--------|---------|
| Cancel a job | `scancel <jobid>` |
| Cancel all your jobs | `scancel --me` |
| Cancel pending jobs only | `scancel --me --state=PENDING` |
| Hold a pending job | `scontrol hold <jobid>` |
| Release a held job | `scontrol release <jobid>` |
| Change time limit (pending) | `scontrol update JobId=<id> TimeLimit=HH:MM:SS` |
| Change partition (pending) | `scontrol update JobId=<id> Partition=<name>` |
| Requeue a job | `scontrol requeue <jobid>` |
| Submit as held | `sbatch --hold script.sh` |

Exercises¶

Submit a long job, then cancel it

Submit a job that sleeps for 1 hour. Confirm it is running or pending with squeue --me, then cancel it with scancel. Verify it is gone from the queue.

Hint / Solution

sbatch --time=01:00:00 --wrap="sleep 3600" --job-name=cancel_test
squeue --me
scancel <jobid>
squeue --me

Hold a pending job and release it

Submit a job in the held state using --hold. Verify its pending reason shows (JobHeldUser) in squeue. Then release it with scontrol release and confirm it returns to normal pending status.

Hint / Solution

sbatch --hold --time=00:10:00 --wrap="echo 'Released and running'" --job-name=hold_test
squeue --me -o "%.8i %.20j %.2t %R"
# Should show PD with reason (JobHeldUser)

scontrol release <jobid>
squeue --me -o "%.8i %.20j %.2t %R"
# Should now show PD with (Priority) or (Resources), or R if running

Modify a pending job's time limit

Submit a job with a 1-hour time limit in held state. Before releasing it, use scontrol update to change the time limit to 2 hours. Verify the change with scontrol show job, then release the job.

Hint / Solution

sbatch --hold --time=01:00:00 --wrap="sleep 60" --job-name=modify_test
scontrol update JobId=<jobid> TimeLimit=02:00:00
scontrol show job <jobid> | grep TimeLimit
# Should show TimeLimit=02:00:00
scontrol release <jobid>

Cancel only pending jobs

Submit three jobs: two short ones that will likely start running and one requesting large resources that will pend. Use scancel with --state=PENDING to cancel only the pending job while leaving the running ones alone.

Hint / Solution

sbatch --time=00:10:00 --wrap="sleep 300" --job-name=runner1
sbatch --time=00:10:00 --wrap="sleep 300" --job-name=runner2
sbatch --time=00:10:00 --mem=500G --wrap="sleep 60" --job-name=will_pend

squeue --me
scancel --me --state=PENDING
squeue --me
# Running jobs should still be there; pending job should be gone

# Clean up
scancel --me

Managing Jobs

Exercises¶

References¶