Submitting Jobs

# Submitting Jobs

## sbatch: The Primary Submission Command

`sbatch` submits a job script to the Slurm queue for later execution on a compute node.

```bash
sbatch [options] script [script_args...]
```

The simplest submission:

```bash
sbatch my_job.sh
```

---

## Job Script Anatomy

A Slurm job script has three parts:

```bash
#!/bin/bash                        # 1. Shebang line (interpreter)

#SBATCH --job-name=my_analysis     # 2. SBATCH directives (resource requests)
#SBATCH --output=%x_%j.out
#SBATCH --error=%x_%j.err
#SBATCH --partition=batch
#SBATCH --time=04:00:00
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=8
#SBATCH --mem=32G

# 3. Your actual work              # Everything below runs on the compute node
module load blast/2.15
cd $SLURM_SUBMIT_DIR

blastn -query input.fasta \
       -db /data/nt \
       -out results.txt \
       -num_threads $SLURM_CPUS_PER_TASK
```

### Rules for #SBATCH Directives

- Must appear **before any executable commands** in the script
- Lines starting with `#SBATCH` are parsed by Slurm, not by the shell
- A blank line or non-`#SBATCH` comment **does not** end the directive block (unlike SGE's `#$`)
- Directives are equivalent to command-line options to `sbatch`

### Command-Line vs. Script Directives

Options can be specified on the command line, in the script, or both. **Command-line options take precedence** over script directives:

```bash
# Script says --time=04:00:00, but command line overrides to 2 hours:
sbatch --time=02:00:00 my_job.sh
```

This is useful for reusing the same script with different parameters.

---

## Common sbatch Options

### Job Identity

| Option | Description | Example |
|--------|-------------|---------|
| `-J, --job-name` | Name for the job | `--job-name=blast_run` |
| `-o, --output` | File for stdout | `--output=logs/%x_%j.out` |
| `-e, --error` | File for stderr (default: same as stdout) | `--error=logs/%x_%j.err` |
| `-A, --account` | Charge to this account | `--account=lab_smith` |

### Resource Requests

| Option | Description | Example |
|--------|-------------|---------|
| `-p, --partition` | Target partition | `--partition=gpu` |
| `-t, --time` | Maximum walltime (HH:MM:SS or D-HH:MM) | `--time=12:00:00` |
| `-n, --ntasks` | Number of tasks (processes) | `--ntasks=4` |
| `-c, --cpus-per-task` | CPUs per task (threads) | `--cpus-per-task=8` |
| `--mem` | Memory per node | `--mem=64G` |
| `--mem-per-cpu` | Memory per CPU | `--mem-per-cpu=4G` |
| `-N, --nodes` | Number of nodes | `--nodes=2` |
| `--gres` | Generic resources (GPUs, etc.) | `--gres=gpu:2` |
| `--exclusive` | Exclusive node access | `--exclusive` |

> **Key difference from SGE:** Slurm jobs generally must specify a time limit (`--time`). If omitted, the partition's default time limit applies, which may be short. Always set `--time` explicitly.

### Notifications

| Option | Description | Example |
|--------|-------------|---------|
| `--mail-type` | When to send email | `--mail-type=BEGIN,END,FAIL` |
| `--mail-user` | Email address | `--mail-user=jdoe@example.com` |

Mail type values: `NONE`, `BEGIN`, `END`, `FAIL`, `REQUEUE`, `ALL`, `TIME_LIMIT_90` (warning at 90% of time limit).

### Working Directory

By default, `sbatch` records the directory where you submit and sets `$SLURM_SUBMIT_DIR` to that path. Your job starts in that directory.

```bash
# Override with:
#SBATCH --chdir=/scratch/jdoe/project
```

---

## Filename Patterns

Output and error filenames support special patterns:

| Pattern | Replaced With | Example Result |
|---------|---------------|---------------|
| `%j` | Job ID | `12345` |
| `%x` | Job name | `blast_run` |
| `%A` | Array job ID | `12340` |
| `%a` | Array task ID | `7` |
| `%N` | First node name | `node042` |
| `%u` | Username | `jdoe` |

```bash
#SBATCH --output=logs/%x_%j.out    # logs/blast_run_12345.out
#SBATCH --error=logs/%x_%j.err     # logs/blast_run_12345.err
```

> **Tip:** Create the `logs/` directory before submitting, or the job will fail.

---

## Passing Arguments to Job Scripts

You can pass arguments to your job script just like a regular shell script:

```bash
$ sbatch my_job.sh input1.fasta /data/output/
```

Inside `my_job.sh`:

```bash
#!/bin/bash
#SBATCH --job-name=process
#SBATCH --time=02:00:00

INPUT=$1     # input1.fasta
OUTDIR=$2    # /data/output/

my_program --input $INPUT --output $OUTDIR
```

---

## Wrapping Simple Commands

For quick one-liners, use `--wrap` instead of writing a script:

```bash
sbatch --time=00:10:00 --mem=4G --wrap="hostname; date; sleep 60"
```

This creates a temporary script internally. Useful for testing, but write a proper script for real work.

---

## Submitting from a Different Directory

```bash
# Submit a script from another location
sbatch /home/jdoe/scripts/my_pipeline.sh

# Override the working directory
sbatch --chdir=/scratch/jdoe/run001 my_pipeline.sh
```

---

## Real-World Examples

### Bioinformatics: BLAST Search

```bash
#!/bin/bash
#SBATCH --job-name=blastn
#SBATCH --output=logs/blast_%j.out
#SBATCH --error=logs/blast_%j.err
#SBATCH --partition=batch
#SBATCH --time=04:00:00
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G

module load blast/2.15

blastn -query $SLURM_SUBMIT_DIR/query.fasta \
       -db /shared/databases/nt \
       -out $SLURM_SUBMIT_DIR/blast_results.txt \
       -evalue 1e-5 \
       -num_threads $SLURM_CPUS_PER_TASK \
       -outfmt 6
```

### Computational Chemistry: Schrodinger Glide Docking

```bash
#!/bin/bash
#SBATCH --job-name=glide_dock
#SBATCH --output=logs/glide_%j.out
#SBATCH --partition=batch
#SBATCH --time=12:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G

module load schrodinger/2026-1

cd $SLURM_SUBMIT_DIR
$SCHRODINGER/glide dock_job.in -NPROC $SLURM_CPUS_PER_TASK -WAIT
```

### GPU Job: AlphaFold Prediction

```bash
#!/bin/bash
#SBATCH --job-name=alphafold
#SBATCH --output=logs/af_%j.out
#SBATCH --partition=gpu
#SBATCH --time=24:00:00
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:1

module load alphafold/2.3

run_alphafold.sh \
    --fasta_paths=$SLURM_SUBMIT_DIR/target.fasta \
    --output_dir=$SLURM_SUBMIT_DIR/af_output \
    --model_preset=monomer
```

---

## What Happens After You Submit

1. Slurm validates your request (partition exists, resources are reasonable, you have permission)
2. Job enters the **pending** queue (state `PD`)
3. The scheduler evaluates pending jobs based on priority, fairshare, and resource availability
4. When resources are available, Slurm **dispatches** your job to a compute node (state `R`)
5. Your script executes on the compute node
6. When the script exits (or time limit is reached), the job **completes** (state `CD`) or **fails** (state `F`)
7. Stdout/stderr are written to your output files
8. Accounting data is recorded in the Slurm database

> **Note:** Jobs are not necessarily dispatched in submission order. Priority, fairshare, and resource requirements all influence scheduling order.

Exercises¶

Submit with a custom output filename

Write a short job script that runs echo "Hello from $HOSTNAME". Use the --output directive to write output to logs/%x_%j.out (job name + job ID). Remember to create the logs/ directory first.

Hint / Solution

mkdir -p logs

cat > custom_output.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=custom_out
#SBATCH --output=logs/%x_%j.out
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --mem=1G

echo "Hello from $HOSTNAME"
EOF

sbatch custom_output.sh

Use --wrap for a one-liner

Submit a quick job using --wrap that prints the current date and the contents of /proc/cpuinfo | head -5. Request 5 minutes and 1G of memory on the command line.

Hint / Solution

sbatch --time=00:05:00 --mem=1G --wrap="date; head -5 /proc/cpuinfo"

Set up email notifications

Modify a job script (or use command-line flags) to send you an email when the job begins, ends, or fails. Submit it and verify the directives with scontrol show job.

Hint / Solution

sbatch --mail-type=BEGIN,END,FAIL --mail-user=you@example.com --time=00:05:00 --wrap="sleep 30"

# Verify the settings
scontrol show job <jobid> | grep -i mail

Pass arguments to a job script

Create a job script called greet.sh that accepts two positional arguments: a name ($1) and a project ($2). Have it print "Hello, [name]! Working on [project]." Submit it with arguments.

Hint / Solution

cat > greet.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=greet
#SBATCH --output=greet_%j.out
#SBATCH --time=00:05:00

NAME=$1
PROJECT=$2
echo "Hello, $NAME! Working on $PROJECT."
EOF

sbatch greet.sh "Alice" "genome_assembly"

Use --chdir to control the working directory

Create a directory called /tmp/$USER/test_chdir. Submit a job that uses --chdir to set the working directory to that path. Inside the script, run pwd to confirm the job started in the right directory.

Hint / Solution

mkdir -p /tmp/$USER/test_chdir

sbatch --chdir=/tmp/$USER/test_chdir --time=00:05:00 --wrap="pwd; echo 'Working dir test' > chdir_test.txt"

# After the job completes, check:
cat /tmp/$USER/test_chdir/chdir_test.txt

Submitting Jobs

Exercises¶

References¶