Job Arrays

<!-- audience: L3 --> <!-- deployment: all --> <!-- sge-source: SGE job arrays + "Effective Usage" --> # Job Arrays ## What Are Job Arrays? Job arrays let you submit **thousands of similar jobs with a single command**. Each task in the array runs the same script but receives a unique index number, which you use to vary input data, parameters, or output filenames. This is the single most important feature for life science HPC workloads, where you often need to run the same analysis on many samples. --- ## Basic Syntax ```bash $ sbatch --array=0-99 my_analysis.sh Submitted batch job 12340 ``` This creates 100 tasks (indices 0 through 99), each running `my_analysis.sh` independently. ### Index Specifications ```bash --array=0-31 # Indices 0, 1, 2, ..., 31 --array=1,3,5,7 # Specific indices --array=1-7:2 # Step size of 2: 1, 3, 5, 7 --array=1-1000%50 # 1-1000, max 50 running simultaneously ``` The `%` throttle is critical for being a good cluster citizen -- it prevents you from monopolizing all available resources. --- ## Environment Variables Inside each array task, Slurm sets these variables: | Variable | Description | Example | |----------|-------------|---------| | `$SLURM_ARRAY_JOB_ID` | The parent job ID | `12340` | | `$SLURM_ARRAY_TASK_ID` | This task's index | `42` | | `$SLURM_ARRAY_TASK_COUNT` | Total number of tasks | `100` | | `$SLURM_ARRAY_TASK_MIN` | Lowest index | `0` | | `$SLURM_ARRAY_TASK_MAX` | Highest index | `99` | Output files use `%A` (array job ID) and `%a` (task ID): ```bash #SBATCH --output=logs/analysis_%A_%a.out #SBATCH --error=logs/analysis_%A_%a.err ``` --- ## Common Patterns ### Pattern 1: Index to Filename Mapping Map each task index to a specific input file: ```bash #!/bin/bash #SBATCH --job-name=blast_array #SBATCH --array=0-49 #SBATCH --output=logs/blast_%A_%a.out #SBATCH --time=02:00:00 #SBATCH --cpus-per-task=4 #SBATCH --mem=8G # Build input filename from array index INPUT_FILE=samples/sample_${SLURM_ARRAY_TASK_ID}.fasta blastn -query $INPUT_FILE \ -db /data/nt \ -out results/result_${SLURM_ARRAY_TASK_ID}.txt \ -num_threads $SLURM_CPUS_PER_TASK ``` ### Pattern 2: File List When filenames don't follow a numeric pattern, use a file list: ```bash #!/bin/bash #SBATCH --job-name=process_samples #SBATCH --array=1-50 #SBATCH --output=logs/%x_%A_%a.out #SBATCH --time=04:00:00 #SBATCH --mem=16G # Read the Nth line from a file list INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" sample_list.txt) echo "Processing: $INPUT" ./analyze.sh "$INPUT" results/ ``` Where `sample_list.txt` contains: ``` /data/samples/patient_A.bam /data/samples/patient_B.bam /data/samples/patient_C.bam ... ``` ### Pattern 3: Parameter Sweep Vary parameters instead of input files: ```bash #!/bin/bash #SBATCH --job-name=param_sweep #SBATCH --array=0-99 #SBATCH --output=logs/sweep_%A_%a.out #SBATCH --time=01:00:00 # Derive parameter from index LEARNING_RATE=$(echo "scale=4; 0.001 * $SLURM_ARRAY_TASK_ID" | bc) python train_model.py \ --learning-rate $LEARNING_RATE \ --output results/model_${SLURM_ARRAY_TASK_ID}.pkl ``` ### Pattern 4: Chunk Processing Process a large file in chunks: ```bash #!/bin/bash #SBATCH --job-name=chunk_process #SBATCH --array=1-10 #SBATCH --output=logs/chunk_%A_%a.out #SBATCH --time=02:00:00 TOTAL_LINES=$(wc -l < big_input.txt) CHUNK_SIZE=$(( (TOTAL_LINES + SLURM_ARRAY_TASK_COUNT - 1) / SLURM_ARRAY_TASK_COUNT )) START=$(( (SLURM_ARRAY_TASK_ID - 1) * CHUNK_SIZE + 1 )) END=$(( SLURM_ARRAY_TASK_ID * CHUNK_SIZE )) sed -n "${START},${END}p" big_input.txt | ./process_lines.sh > results/chunk_${SLURM_ARRAY_TASK_ID}.out ``` --- ## Managing Array Jobs ### Check Status ```bash # Overview (tasks are grouped) $ squeue --me JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 12340_[50-99] batch blast jdoe PD 0:00 1 (Resources) 12340_42 batch blast jdoe R 0:05 1 node015 12340_43 batch blast jdoe R 0:03 1 node022 # Show each task separately $ squeue --me -r ``` ### Cancel Specific Tasks ```bash # Cancel one task $ scancel 12340_42 # Cancel a range of tasks $ scancel 12340_[50-99] # Cancel all tasks $ scancel 12340 ``` ### Check Completed Array Job Results ```bash # Summary of all tasks $ sacct -j 12340 --format=JobID,Elapsed,State,ExitCode JobID Elapsed State ExitCode --------- ------- --------- -------- 12340_0 0:23:45 COMPLETED 0:0 12340_1 0:22:12 COMPLETED 0:0 12340_2 0:24:01 FAILED 1:0 ... # Find failed tasks $ sacct -j 12340 --state=FAILED --format=JobID,State,ExitCode ``` --- ## Throttling: Be a Good Neighbor Without throttling, a 10,000-task array can consume every available slot on the cluster. Use `%` to limit concurrent tasks: ```bash #SBATCH --array=1-10000%100 # Max 100 tasks running at once ``` Guidelines: - Small cluster (<100 nodes): `%20` to `%50` - Medium cluster (100-500 nodes): `%50` to `%200` - Always consider other users and fairshare policies --- ## Array Jobs vs. Submission Loops **Don't do this:** ```bash # BAD: submitting 1000 individual jobs in a loop for i in $(seq 1 1000); do sbatch my_job.sh $i done ``` **Do this instead:** ```bash # GOOD: one array job sbatch --array=1-1000%50 my_job.sh ``` Why arrays are better: - Submitted in milliseconds vs. seconds per job - Much less load on the scheduler - Easier to manage (one job ID to track) - Better scheduling (Slurm understands the relationship)

Exercises

1. Submit a 10-task array job

Write a job script that prints "Task [index] running on [hostname]" for each array task. Submit it with --array=1-10. After all tasks complete, check that you have 10 output files.

Hint / Solution

cat > array_test.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=array_test
#SBATCH --output=array_%A_%a.out
#SBATCH --time=00:05:00
#SBATCH --mem=1G

echo "Task $SLURM_ARRAY_TASK_ID running on $(hostname)"
EOF

sbatch --array=1-10 array_test.sh

# After completion:
ls array_*.out | wc -l
# Should show 10

1. Use SLURM_ARRAY_TASK_ID to process different input files

Create 5 small input files (sample_1.txt through sample_5.txt), each containing a unique line of text. Write an array job that reads the file corresponding to its task ID and prints the contents.

Hint / Solution

# Create sample input files
for i in $(seq 1 5); do
    echo "Data for sample $i" > sample_${i}.txt
done

cat > process_samples.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=process_samples
#SBATCH --output=logs/sample_%A_%a.out
#SBATCH --time=00:05:00
#SBATCH --mem=1G

INPUT=sample_${SLURM_ARRAY_TASK_ID}.txt
echo "Processing $INPUT:"
cat $INPUT
EOF

mkdir -p logs
sbatch --array=1-5 process_samples.sh

1. Throttle an array to 3 concurrent tasks

Submit a 10-task array job where each task sleeps for 60 seconds, but limit concurrency to at most 3 tasks running at the same time. While it runs, use squeue --me to verify that no more than 3 tasks are in R state.

Hint / Solution

sbatch --array=1-10%3 --time=00:05:00 --wrap="sleep 60" --job-name=throttle_test

# Check that at most 3 are running
squeue --me

1. Cancel a single array task

Submit a 10-task array job where each task sleeps for 5 minutes. Once tasks are running, cancel only task 5. Verify with squeue that the other tasks are still active.

Hint / Solution

sbatch --array=1-10 --time=00:10:00 --wrap="sleep 300" --job-name=cancel_one

# Note the array job ID (e.g., 12340)
squeue --me -r

# Cancel only task 5
scancel 12340_5

# Verify task 5 is gone but others remain
squeue --me -r

# Clean up
scancel 12340

1. Use a file list with an array job

Create a file called file_list.txt with 4 file paths (one per line). Write an array job with --array=1-4 that reads the Nth line from file_list.txt using sed and prints it. This pattern is essential for processing samples with non-numeric names.

Hint / Solution

cat > file_list.txt << 'EOF'
/data/samples/patient_alpha.bam
/data/samples/patient_beta.bam
/data/samples/patient_gamma.bam
/data/samples/patient_delta.bam
EOF

cat > filelist_array.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=filelist
#SBATCH --output=filelist_%A_%a.out
#SBATCH --time=00:05:00
#SBATCH --mem=1G

INPUT=$(sed -n "${SLURM_ARRAY_TASK_ID}p" file_list.txt)
echo "Task $SLURM_ARRAY_TASK_ID would process: $INPUT"
EOF

sbatch --array=1-4 filelist_array.sh

References

• SchedMD: Job Array Support
• SchedMD: sbatch --array option