Managing Jobs

<!-- audience: L2 --> <!-- deployment: all --> <!-- sge-source: SGE "Job Control" --> # Managing Jobs ## scancel: Canceling Jobs ### Cancel a Specific Job ```bash $ scancel 12345 ``` ### Cancel All Your Jobs ```bash $ scancel --me ``` ### Cancel by Name ```bash $ scancel --name=blast_run ``` ### Cancel by State ```bash # Cancel all your pending jobs (keep running ones) $ scancel --me --state=PENDING # Cancel all your jobs in a specific partition $ scancel --me --partition=gpu ``` --- ## scancel: Advanced Options ### Cancel Specific Array Tasks ```bash # Cancel task 5 of array job 12340 $ scancel 12340_5 # Cancel tasks 1-10 $ scancel 12340_[1-10] ``` ### Send a Signal Instead of Killing By default, `scancel` sends `SIGKILL`. To give your job a chance to clean up: ```bash # Send SIGTERM first (allows graceful shutdown) $ scancel --signal=TERM 12345 # After a delay, follow up with kill if needed $ scancel 12345 ``` > **ParallelCluster Note:** Dynamic compute nodes that become idle are automatically terminated by the cluster autoscaler. Canceling a job on a dynamic node may trigger node termination if no other jobs are queued for that node. > **PCS Note:** Node lifecycle is managed by PCS. Canceled jobs release resources back to the managed pool, and idle nodes scale down based on your queue's scaling configuration. --- ## scontrol hold / release: Pausing Job Dispatch You can prevent a pending job from being scheduled without canceling it: ### Hold a Job ```bash $ scontrol hold 12345 ``` The job stays in the queue but won't be dispatched until released. State shows as `PD` with reason `(JobHeldUser)`. ### Release a Held Job ```bash $ scontrol release 12345 ``` The job returns to normal pending status and will be scheduled when resources are available. > **Use case:** You submitted a job but realized you need to fix something in the input files. Hold the job, fix the files, then release it -- no need to cancel and resubmit. --- ## scontrol update: Modifying Pending Jobs You can modify certain attributes of a **pending** job (not running jobs): ### Change Time Limit ```bash $ scontrol update JobId=12345 TimeLimit=8:00:00 ``` ### Change Partition ```bash $ scontrol update JobId=12345 Partition=highmem ``` ### Change Job Name ```bash $ scontrol update JobId=12345 JobName=new_name ``` ### Change Account ```bash $ scontrol update JobId=12345 Account=other_lab ``` ### Change Resource Requests ```bash # Change number of CPUs $ scontrol update JobId=12345 NumCPUs=32 # Change memory $ scontrol update JobId=12345 MinMemoryNode=64G ``` > **Limitation:** Once a job is running, most attributes cannot be changed. The time limit is an exception -- administrators can extend it for running jobs. --- ## scontrol requeue: Restarting Jobs If a job needs to be rerun (e.g., it was affected by a node failure): ```bash $ scontrol requeue 12345 ``` This cancels the running job and puts it back in the pending queue. The job must have `--requeue` enabled (which is the default on most clusters). --- ## Practical Scenarios ### "I submitted the wrong script" ```bash # Cancel and resubmit $ scancel 12345 $ sbatch corrected_script.sh ``` ### "I need more time" ```bash # For a pending job, update directly $ scontrol update JobId=12345 TimeLimit=24:00:00 # For a running job, ask your admin (users typically can't extend running jobs) ``` ### "I want to submit jobs but not start them yet" ```bash # Submit in held state $ sbatch --hold my_job.sh Submitted batch job 12346 # Later, release when ready $ scontrol release 12346 ``` --- ## Practical Scenarios (cont.) ### "I need to cancel everything and start over" ```bash # Cancel all your jobs $ scancel --me # Verify nothing is left $ squeue --me # (should show no jobs) ``` ### "My job is pending forever" ```bash # Check why $ squeue --me -o "%.8i %.9P %.20j %.2t %R" JOBID PARTITION NAME ST REASON 12347 gpu alphafold PD (Resources) # Get more details $ scontrol show job 12347 | grep -i reason Reason=Resources # Check partition availability $ sinfo -p gpu PARTITION AVAIL TIMELIMIT NODES STATE NODELIST gpu up 2-00:00:00 8 alloc gpu[01-08] # All GPU nodes are in use -- you need to wait ``` --- ## Job Lifecycle Summary ``` sbatch ──> PENDING ──> RUNNING ──> COMPLETED | | | hold/release scancel (success) | | (JobHeldUser) CANCELLED | requeue ──> PENDING | FAILED / TIMEOUT / OOM ``` --- ## Command Quick Reference | Action | Command | |--------|---------| | Cancel a job | `scancel <jobid>` | | Cancel all your jobs | `scancel --me` | | Cancel pending jobs only | `scancel --me --state=PENDING` | | Hold a pending job | `scontrol hold <jobid>` | | Release a held job | `scontrol release <jobid>` | | Change time limit (pending) | `scontrol update JobId=<id> TimeLimit=HH:MM:SS` | | Change partition (pending) | `scontrol update JobId=<id> Partition=<name>` | | Requeue a job | `scontrol requeue <jobid>` | | Submit as held | `sbatch --hold script.sh` |

Exercises

1. Submit a long job, then cancel it

Submit a job that sleeps for 1 hour. Confirm it is running or pending with squeue --me, then cancel it with scancel. Verify it is gone from the queue.

Hint / Solution

sbatch --time=01:00:00 --wrap="sleep 3600" --job-name=cancel_test
squeue --me
scancel <jobid>
squeue --me

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --time=01:00:00 --job-name=val_mg1 --output=/dev/null --wrap='sleep 3600'
Submitted batch job 42

$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               42     batch  val_mg1 ec2-user  R       0:02      1 batch-dy-compute-1

$ scancel 42

$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

1. Hold a pending job and release it

Submit a job in the held state using --hold. Verify its pending reason shows (JobHeldUser) in squeue. Then release it with scontrol release and confirm it returns to normal pending status.

Hint / Solution

sbatch --hold --time=00:10:00 --wrap="echo 'Released and running'" --job-name=hold_test
squeue --me -o "%.8i %.20j %.2t %R"
# Should show PD with reason (JobHeldUser)

scontrol release <jobid>
squeue --me -o "%.8i %.20j %.2t %R"
# Should now show PD with (Priority) or (Resources), or R if running

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --hold --time=00:10:00 --job-name=val_mg2 --output=/dev/null --wrap='echo released'
Submitted batch job 42

$ squeue --me -o '%.8i %.20j %.2t %R'
   JOBID                 NAME ST NODELIST(REASON)
     42              val_mg2 PD (JobHeldUser)

$ scontrol release 42

$ squeue --me -o '%.8i %.20j %.2t %R'
   JOBID                 NAME ST NODELIST(REASON)

1. Modify a pending job's time limit

Submit a job with a 1-hour time limit in held state. Before releasing it, use scontrol update to change the time limit to 2 hours. Verify the change with scontrol show job, then release the job.

Hint / Solution

sbatch --hold --time=01:00:00 --wrap="sleep 60" --job-name=modify_test
scontrol update JobId=<jobid> TimeLimit=02:00:00
scontrol show job <jobid> | grep TimeLimit
# Should show TimeLimit=02:00:00
scontrol release <jobid>

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --hold --time=01:00:00 --job-name=val_mg3 --output=/dev/null --wrap='sleep 60'
Submitted batch job 42

$ scontrol update JobId=42 TimeLimit=02:00:00

$ scontrol show job 42 | grep TimeLimit
   RunTime=00:00:00 TimeLimit=02:00:00 TimeMin=N/A

$ scontrol release 42

1. Cancel only pending jobs

Submit three jobs: two short ones that will likely start running and one requesting large resources that will pend. Use scancel with --state=PENDING to cancel only the pending job while leaving the running ones alone.

Hint / Solution

sbatch --time=00:10:00 --wrap="sleep 300" --job-name=runner1
sbatch --time=00:10:00 --wrap="sleep 300" --job-name=runner2
sbatch --time=00:10:00 --mem=500G --wrap="sleep 60" --job-name=will_pend

squeue --me
scancel --me --state=PENDING
squeue --me
# Running jobs should still be there; pending job should be gone

# Clean up
scancel --me

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch --time=00:05:00 --job-name=val_mg4a --output=/dev/null --wrap='sleep 60'
Submitted batch job 42

$ sbatch --time=00:05:00 --job-name=val_mg4b --output=/dev/null --wrap='sleep 60'
Submitted batch job 43

$ sbatch --hold --time=00:05:00 --job-name=val_mg4p --output=/dev/null --wrap='sleep 60'
Submitted batch job 44

$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               44     batch val_mg4p ec2-user PD       0:00      1 (JobHeldUser)
               42     batch val_mg4a ec2-user  R       0:01      1 batch-dy-compute-1
               43     batch val_mg4b ec2-user  R       0:01      1 batch-dy-compute-2

$ scancel --me --state=PENDING

$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
               42     batch val_mg4a ec2-user  R       0:03      1 batch-dy-compute-1
               43     batch val_mg4b ec2-user  R       0:03      1 batch-dy-compute-2

$ scancel --me

References

• SchedMD: scancel man page
• SchedMD: scontrol man page
• SchedMD: Job Reason Codes