Job Dependencies

<!-- audience: L3 --> <!-- deployment: all --> <!-- sge-source: minimal SGE coverage, expanded for Slurm --> # Job Dependencies ## Why Dependencies? Scientific workflows often involve sequential steps: preprocess data, run analysis, generate reports. Job dependencies let you build pipelines where later jobs automatically wait for earlier ones to finish. --- ## Basic Syntax ```bash sbatch --dependency=<type>:<jobid>[:<jobid>...] script.sh ``` --- ## Dependency Types | Type | Job starts when... | |------|-------------------| | `afterok:jobid` | Job `jobid` completed **successfully** (exit code 0) | | `afternotok:jobid` | Job `jobid` **failed** (non-zero exit code) | | `afterany:jobid` | Job `jobid` finished (regardless of success/failure) | | `after:jobid` | Job `jobid` has **started** (or been cancelled) | | `aftercorr:jobid` | Corresponding task of array job `jobid` completed successfully | | `singleton` | No other job with the same name and user is running | --- ## Building a Simple Pipeline ```bash # Step 1: Preprocessing $ JOB1=$(sbatch --parsable preprocess.sh) echo "Preprocessing: $JOB1" # Step 2: Analysis (runs after preprocessing succeeds) $ JOB2=$(sbatch --parsable --dependency=afterok:$JOB1 analyze.sh) echo "Analysis: $JOB2" # Step 3: Report (runs after analysis succeeds) $ JOB3=$(sbatch --parsable --dependency=afterok:$JOB2 report.sh) echo "Report: $JOB3" ``` The `--parsable` flag makes `sbatch` output only the job ID, which makes it easy to capture in a variable. ### Check the Chain ```bash $ squeue --me -o "%.8i %.20j %.2t %E" JOBID NAME ST DEPENDENCY 12345 preprocess R 12346 analyze PD afterok:12345 12347 report PD afterok:12346 ``` --- ## Multiple Dependencies ### AND Logic (comma-separated) All dependencies must be satisfied: ```bash # Wait for BOTH jobs to complete successfully $ sbatch --dependency=afterok:12345,afterok:12346 merge.sh ``` ### OR Logic (question-mark-separated) Any dependency being satisfied is enough: ```bash # Start if EITHER job succeeds $ sbatch --dependency=afterok:12345?afterok:12346 next_step.sh ``` --- ## Practical Pipeline Example A typical genomics pipeline: align, sort, index, call variants, filter: ```bash #!/bin/bash # submit_pipeline.sh - Submit a complete variant calling pipeline SAMPLE=$1 echo "Submitting pipeline for sample: $SAMPLE" # Step 1: Align reads ALIGN=$(sbatch --parsable --job-name=align_${SAMPLE} \ --time=04:00:00 --cpus-per-task=16 --mem=32G \ align.sh $SAMPLE) echo " Align: $ALIGN" # Step 2: Sort BAM (after alignment) SORT=$(sbatch --parsable --job-name=sort_${SAMPLE} \ --dependency=afterok:$ALIGN \ --time=02:00:00 --cpus-per-task=4 --mem=16G \ sort_bam.sh $SAMPLE) echo " Sort: $SORT" # Step 3: Index BAM (after sorting) INDEX=$(sbatch --parsable --job-name=index_${SAMPLE} \ --dependency=afterok:$SORT \ --time=00:30:00 --mem=4G \ index_bam.sh $SAMPLE) echo " Index: $INDEX" # Step 4: Call variants (after indexing) CALL=$(sbatch --parsable --job-name=call_${SAMPLE} \ --dependency=afterok:$INDEX \ --time=08:00:00 --cpus-per-task=4 --mem=16G \ call_variants.sh $SAMPLE) echo " Call: $CALL" # Step 5: Filter (after calling) FILTER=$(sbatch --parsable --job-name=filter_${SAMPLE} \ --dependency=afterok:$CALL \ --time=01:00:00 --mem=8G \ filter_variants.sh $SAMPLE) echo " Filter: $FILTER" echo "Pipeline submitted. Final job: $FILTER" ``` Usage: ```bash $ bash submit_pipeline.sh NA12878 Submitting pipeline for sample: NA12878 Align: 12350 Sort: 12351 Index: 12352 Call: 12353 Filter: 12354 Pipeline submitted. Final job: 12354 ``` --- ## Fan-Out / Fan-In Pattern Process multiple samples in parallel, then merge: ```bash #!/bin/bash # submit_multi_sample.sh MERGE_DEPS="" for SAMPLE in sampleA sampleB sampleC sampleD; do JOB=$(sbatch --parsable --job-name=process_${SAMPLE} \ --time=04:00:00 --mem=16G \ process.sh $SAMPLE) # Build dependency list for merge step if [ -z "$MERGE_DEPS" ]; then MERGE_DEPS="afterok:$JOB" else MERGE_DEPS="${MERGE_DEPS},afterok:$JOB" fi done # Merge runs after ALL samples complete MERGE=$(sbatch --parsable --job-name=merge_all \ --dependency=$MERGE_DEPS \ --time=02:00:00 --mem=32G \ merge_results.sh) echo "Merge job: $MERGE (waiting on: $MERGE_DEPS)" ``` --- ## Array Job Dependencies Use `aftercorr` to chain array jobs where task N of the second array waits for task N of the first: ```bash $ JOB1=$(sbatch --parsable --array=1-100 step1.sh) $ JOB2=$(sbatch --parsable --array=1-100 --dependency=aftercorr:$JOB1 step2.sh) ``` Task `JOB2_5` won't start until `JOB1_5` completes successfully. --- ## Singleton Dependencies Ensure only one instance of a named job runs at a time: ```bash #SBATCH --job-name=daily_backup #SBATCH --dependency=singleton ``` This is useful for recurring jobs (e.g., database updates) where concurrent execution would cause conflicts. --- ## Handling Failures When a dependency can't be satisfied (e.g., the parent job failed and you used `afterok`), the dependent job is marked with state `PD` and reason `(DependencyNeverSatisfied)`. To handle failures gracefully: ```bash # Use afterany instead of afterok for cleanup jobs $ sbatch --dependency=afterany:$JOB1 cleanup.sh # Or submit an error-handling job $ sbatch --dependency=afternotok:$JOB1 handle_error.sh ```

Exercises

1. Create a 2-job pipeline with afterok

Write two simple scripts: step1.sh that creates a file called step1_done.txt, and step2.sh that checks for that file and prints "Pipeline complete." Submit them so that step 2 only runs after step 1 succeeds. Use --parsable to capture the job ID.

Hint / Solution

cat > step1.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=step1
#SBATCH --output=step1_%j.out
#SBATCH --time=00:05:00

echo "Step 1 running"
touch step1_done.txt
EOF

cat > step2.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=step2
#SBATCH --output=step2_%j.out
#SBATCH --time=00:05:00

if [ -f step1_done.txt ]; then
    echo "Pipeline complete. step1_done.txt exists."
else
    echo "ERROR: step1_done.txt not found"
    exit 1
fi
EOF

JOB1=$(sbatch --parsable step1.sh)
JOB2=$(sbatch --parsable --dependency=afterok:$JOB1 step2.sh)
echo "Step 1: $JOB1, Step 2: $JOB2"

# Verify the dependency chain
squeue --me -o "%.8i %.20j %.2t %E"

1. Fan-in: Job C depends on both A and B

Submit two independent jobs (A and B) that each sleep for 30 seconds. Then submit job C with a dependency on both A and B succeeding. Verify with squeue that C shows both dependencies.

Hint / Solution

JOBA=$(sbatch --parsable --time=00:05:00 --job-name=jobA --wrap="sleep 30; echo 'A done'")
JOBB=$(sbatch --parsable --time=00:05:00 --job-name=jobB --wrap="sleep 30; echo 'B done'")
JOBC=$(sbatch --parsable --dependency=afterok:${JOBA},afterok:${JOBB} \
    --time=00:05:00 --job-name=jobC --wrap="echo 'C: both A and B finished'")

echo "A=$JOBA B=$JOBB C=$JOBC"
squeue --me -o "%.8i %.20j %.2t %E"

1. Use singleton dependency

Submit two jobs with the same job name and --dependency=singleton. The second job should wait until the first one completes before starting. Verify by checking squeue to see the second job pending with reason (Dependency).

Hint / Solution

sbatch --time=00:05:00 --job-name=singleton_test --wrap="echo 'First run'; sleep 60"
sbatch --time=00:05:00 --job-name=singleton_test --dependency=singleton --wrap="echo 'Second run'"

squeue --me -o "%.8i %.20j %.2t %R"
# Second job should show PD with (Dependency)

1. Handle a failed dependency

Submit a job (step 1) that intentionally fails (exit 1). Then submit a step 2 with afterok dependency and a separate error handler with afternotok dependency. Observe that step 2 never runs (state DependencyNeverSatisfied) while the error handler does run.

Hint / Solution

JOB1=$(sbatch --parsable --time=00:05:00 --job-name=will_fail --wrap="echo 'Failing now'; exit 1")
JOB2=$(sbatch --parsable --dependency=afterok:$JOB1 --time=00:05:00 \
    --job-name=next_step --wrap="echo 'This should not run'")
JOB3=$(sbatch --parsable --dependency=afternotok:$JOB1 --time=00:05:00 \
    --job-name=error_handler --wrap="echo 'Handling failure from job $JOB1'")

echo "Fail=$JOB1 Next=$JOB2 Handler=$JOB3"

# After JOB1 fails, check:
squeue --me -o "%.8i %.20j %.2t %R"
# JOB2 should show DependencyNeverSatisfied; JOB3 should run

# Clean up JOB2 since it will never run
scancel $JOB2

References

• SchedMD: sbatch --dependency option
• SchedMD: Job Array Support