Recurring Jobs (scrontab)

<!-- audience: L3 --> <!-- deployment: all --> <!-- sge-source: not in SGE materials, Slurm-specific feature --> # Recurring Jobs with scrontab ## What is scrontab? `scrontab` is Slurm's built-in cron-like mechanism for scheduling recurring batch jobs. It uses familiar crontab syntax but runs jobs through the Slurm scheduler, giving you proper resource management, accounting, and fairshare integration. --- ## Basic Usage ```bash # Edit your scrontab (opens in $EDITOR) $ scrontab -e # List your scrontab $ scrontab -l # Remove your scrontab $ scrontab -r ``` --- ## Syntax `scrontab` uses standard cron time fields plus `#SCRON` directives for Slurm options: ``` # ┌───────────── minute (0 - 59) # │ ┌───────────── hour (0 - 23) # │ │ ┌───────────── day of month (1 - 31) # │ │ │ ┌───────────── month (1 - 12) # │ │ │ │ ┌───────────── day of week (0 - 7, 0 and 7 are Sunday) # │ │ │ │ │ # * * * * * command ``` --- ## Examples ### Run a Job Every Hour ``` #SCRON --time=00:30:00 #SCRON --mem=4G @hourly /home/jdoe/scripts/hourly_check.sh ``` ### Run a Daily Backup at 2 AM ``` #SCRON --time=02:00:00 #SCRON --mem=16G #SCRON --partition=batch #SCRON -J daily_backup 0 2 * * * /home/jdoe/scripts/backup_results.sh ``` ### Process New Data Every Weekday at 8 AM ``` #SCRON --time=04:00:00 #SCRON --cpus-per-task=8 #SCRON --mem=32G #SCRON -J morning_pipeline 0 8 * * 1-5 /home/jdoe/pipelines/process_new_data.sh ``` ### Run Every 5 Minutes During Work Hours ``` #SCRON --time=00:05:00 #SCRON --mem=1G 1-5,30 8-17/2 * * mon-fri /home/jdoe/scripts/monitor_queue.sh ``` ### Using Variables ``` DIR=/home/jdoe/scripts #SCRON --time=01:00:00 #SCRON --mem=8G 0 6 * * * $DIR/morning_report.sh #SCRON --time=02:00:00 #SCRON --mem=16G 0 22 * * * $DIR/nightly_cleanup.sh ``` --- ## Supported #SCRON Options `#SCRON` lines accept most `sbatch` options: ``` #SCRON --partition=batch #SCRON --time=01:00:00 #SCRON --cpus-per-task=4 #SCRON --mem=8G #SCRON --gres=gpu:1 #SCRON -J my_job_name #SCRON -A my_account #SCRON --output=/home/jdoe/logs/cron_%j.out #SCRON --error=/home/jdoe/logs/cron_%j.err ``` `#SCRON` directives reset between each crontab entry, so you can set different resources for different jobs. > **ParallelCluster Note:** scrontab works on ParallelCluster because the head node (and `slurmctld`) runs persistently. However, if the head node is stopped or terminated, pending scrontab jobs will not fire until it is restarted. > **PCS Note:** scrontab availability depends on how AWS manages the Slurm control plane. Verify with your administrator that the managed `slurmctld` supports scrontab before relying on it for critical recurring workflows. --- ## Important Behavior - **No guaranteed exact timing.** Jobs are subject to normal Slurm queuing. The cron expression sets the *earliest* eligible start time, but actual start may be delayed. - **Sequential execution.** The next occurrence won't be submitted until the previous one completes. If a job runs past its next scheduled time, that occurrence is skipped. - **Single job ID.** Each crontab entry gets one persistent job ID. Canceling it (`scancel`) stops all future runs and comments out the entry in your scrontab. - **Requeue skips.** `scontrol requeue <jobid>` skips the current occurrence and reschedules for the next valid time. --- ## Use Cases for Life Science | Use Case | Schedule | Example | |----------|----------|---------| | Monitor data transfer progress | Every 30 min | `*/30 * * * *` | | Process new sequencing data | Daily at 6 AM | `0 6 * * *` | | Generate weekly utilization reports | Monday 8 AM | `0 8 * * 1` | | Clean up old temp files | Sunday 2 AM | `0 2 * * 0` | | Sync database updates | Every 6 hours | `0 */6 * * *` | --- ## Viewing and Managing ```bash # List your crontab entries $ scrontab -l # Check if the cron job is in the queue $ squeue --me --name=daily_backup # View history of cron job runs $ sacct --name=daily_backup --starttime=now-7days ```

Exercises

1. Create a scrontab entry that runs every hour

Use scrontab -e to add an entry that runs a simple script every hour. The script should append the current date and hostname to a log file. Set resource limits of 10 minutes and 1G memory.

Hint / Solution

First create the script:

cat > ~/scripts/hourly_check.sh << 'EOF'
#!/bin/bash
echo "$(date): Running on $(hostname)" >> ~/logs/hourly_check.log
EOF
chmod +x ~/scripts/hourly_check.sh
mkdir -p ~/logs

Then edit your scrontab with `scrontab -e` and add:

#SCRON --time=00:10:00
#SCRON --mem=1G
#SCRON -J hourly_check
#SCRON --output=/home/jdoe/logs/hourly_%j.out
@hourly /home/jdoe/scripts/hourly_check.sh

Use absolute paths for all files in scrontab entries.

1. List your scrontab entries

After creating an entry, verify it was saved by listing your scrontab. Also use squeue to find the associated job ID that Slurm created for your recurring job.

Hint / Solution

# List your scrontab
scrontab -l

# Find the cron job in the queue
squeue --me --name=hourly_check

1. Remove a scrontab entry

Remove all your scrontab entries using scrontab -r. Verify the removal by listing again and checking that the associated job is no longer in the queue.

Hint / Solution

# Remove all entries
scrontab -r

# Verify removal
scrontab -l
# Should show an empty or commented-out scrontab

squeue --me --name=hourly_check
# Should return no results

References

• SchedMD: scrontab man page