PBS to Slurm

# PBS/Torque to Slurm Migration Guide

This guide is for users and administrators transitioning from PBS Pro, OpenPBS, or Torque to Slurm.

## Command Mapping

| Action | PBS/Torque | Slurm | Notes |
|--------|-----------|-------|-------|
| Submit batch job | `qsub script.sh` | `sbatch script.sh` | |
| Delete job | `qdel job_id` | `scancel job_id` | |
| Job status | `qstat` | `squeue` | |
| My jobs | `qstat -u $USER` | `squeue --me` | |
| Detailed job info | `qstat -f job_id` | `scontrol show job job_id` | |
| Completed job info | `tracejob job_id` | `sacct -j job_id` | |
| Queue info | `qstat -Q` | `sinfo` | |
| Full queue info | `qstat -Qf` | `scontrol show partition` | |
| Node status | `pbsnodes -l` | `sinfo -N -l` | |
| Specific node | `pbsnodes node001` | `scontrol show node node001` | |
| Interactive job | `qsub -I` | `srun --pty bash` | |
| Interactive with X11 | `qsub -I -X` | `srun --pty --x11 bash` | |
| Modify pending job | `qalter job_id` | `scontrol update JobId=job_id` | |
| Hold job | `qhold job_id` | `scontrol hold job_id` | |
| Release job | `qrls job_id` | `scontrol release job_id` | |
| Move job to queue | `qmove queue job_id` | `scontrol update JobId=job_id Partition=part` | |
| Server status | `qstat -Bf` | `scontrol show config` | |

## Job Script Directive Mapping

| PBS Directive | Slurm Directive | Description |
|--------------|-----------------|-------------|
| `#PBS -N name` | `#SBATCH --job-name=name` | Job name |
| `#PBS -o path` | `#SBATCH --output=path` | Stdout file |
| `#PBS -e path` | `#SBATCH --error=path` | Stderr file |
| `#PBS -j oe` | `#SBATCH --output=file` | Merge stdout/stderr (Slurm default) |
| `#PBS -q queue` | `#SBATCH --partition=partition` | Queue/partition |
| `#PBS -l walltime=HH:MM:SS` | `#SBATCH --time=HH:MM:SS` | Walltime |
| `#PBS -l nodes=N:ppn=P` | `#SBATCH --nodes=N --ntasks-per-node=P` | Nodes and procs per node |
| `#PBS -l select=N:ncpus=C:mem=M` | `#SBATCH --nodes=N --cpus-per-task=C --mem=M` | Resource chunk (PBS Pro) |
| `#PBS -l mem=8gb` | `#SBATCH --mem=8G` | Total memory |
| `#PBS -l pmem=4gb` | `#SBATCH --mem-per-cpu=4G` | Per-process memory |
| `#PBS -l ngpus=1` | `#SBATCH --gres=gpu:1` | GPU request |
| `#PBS -J 1-100` | `#SBATCH --array=1-100` | Job array (PBS Pro) |
| `#PBS -t 1-100` | `#SBATCH --array=1-100` | Job array (Torque) |
| `#PBS -A account` | `#SBATCH --account=account` | Account/project |
| `#PBS -M email` | `#SBATCH --mail-user=email` | Email address |
| `#PBS -m abe` | `#SBATCH --mail-type=BEGIN,END,FAIL` | Email events |
| `#PBS -W depend=afterok:id` | `#SBATCH --dependency=afterok:id` | Dependency |
| `#PBS -l feature=gpu` | `#SBATCH --constraint=gpu` | Node feature |
| `#PBS -V` | `#SBATCH --export=ALL` | Export environment (Slurm default) |
| `#PBS -d /path` | `#SBATCH --chdir=/path` | Working directory |
| `#PBS -r y` | `#SBATCH --requeue` | Requeue on failure |

## Environment Variable Mapping

| PBS Variable | Slurm Variable | Description |
|-------------|----------------|-------------|
| `$PBS_JOBID` | `$SLURM_JOB_ID` | Job ID |
| `$PBS_JOBNAME` | `$SLURM_JOB_NAME` | Job name |
| `$PBS_ARRAY_INDEX` | `$SLURM_ARRAY_TASK_ID` | Array task index |
| `$PBS_NODEFILE` | `$SLURM_JOB_NODELIST` | Allocated nodes (format differs) |
| `$PBS_NP` | `$SLURM_NTASKS` | Total processor count |
| `$PBS_NUM_NODES` | `$SLURM_JOB_NUM_NODES` | Number of nodes |
| `$PBS_NUM_PPN` | `$SLURM_NTASKS_PER_NODE` | Procs per node |
| `$PBS_O_WORKDIR` | `$SLURM_SUBMIT_DIR` | Submission directory |
| `$PBS_O_HOME` | `$HOME` | Home directory |
| `$PBS_QUEUE` | `$SLURM_JOB_PARTITION` | Queue/partition |
| `$TMPDIR` | `$TMPDIR` | Temp directory (site-dependent) |

## Job Script Translation: Side by Side

### PBS Version

```bash
#!/bin/bash
#PBS -N genome_align
#PBS -o logs/align.out
#PBS -e logs/align.err
#PBS -q batch
#PBS -l walltime=08:00:00
#PBS -l nodes=1:ppn=16
#PBS -l mem=32gb
#PBS -V

cd $PBS_O_WORKDIR
module load bwa/0.7.17 samtools/1.17

bwa mem -t $PBS_NUM_PPN \
    /data/reference/hg38.fa \
    reads_R1.fastq.gz reads_R2.fastq.gz \
    | samtools sort -@ 4 -o aligned.bam

samtools index aligned.bam
```

### Slurm Version

```bash
#!/bin/bash
#SBATCH --job-name=genome_align
#SBATCH --output=logs/align.out
#SBATCH --error=logs/align.err
#SBATCH --partition=batch
#SBATCH --time=08:00:00
#SBATCH --nodes=1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G

module purge
module load bwa/0.7.17 samtools/1.17

bwa mem -t $SLURM_CPUS_PER_TASK \
    /data/reference/hg38.fa \
    reads_R1.fastq.gz reads_R2.fastq.gz \
    | samtools sort -@ 4 -o aligned.bam

samtools index aligned.bam
```

### Key Changes

1. `#PBS` becomes `#SBATCH`
2. `$PBS_O_WORKDIR` becomes `$SLURM_SUBMIT_DIR` (but `cd` is usually not needed -- Slurm starts in the submit dir by default)
3. `$PBS_NUM_PPN` becomes `$SLURM_CPUS_PER_TASK`
4. `-l nodes=1:ppn=16` becomes `--nodes=1 --cpus-per-task=16` (for threads) or `--ntasks-per-node=16` (for MPI)
5. `-l mem=32gb` becomes `--mem=32G` (note: `G` not `gb`)
6. `-V` is default in Slurm (environment is exported automatically)

### MPI Job Translation

**PBS:**
```bash
#!/bin/bash
#PBS -N mpi_job
#PBS -l walltime=24:00:00
#PBS -l nodes=4:ppn=32
#PBS -q batch

cd $PBS_O_WORKDIR
module load openmpi/4.1

mpirun -np $PBS_NP ./my_mpi_app
```

**Slurm:**
```bash
#!/bin/bash
#SBATCH --job-name=mpi_job
#SBATCH --time=24:00:00
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=32
#SBATCH --partition=batch

module purge
module load openmpi/4.1

srun ./my_mpi_app
```

Note: `srun` replaces `mpirun` and automatically knows the task count from the `#SBATCH` directives.

## Concept Mapping

| PBS Concept | Slurm Equivalent | Key Differences |
|-------------|------------------|-----------------|
| Queue | Partition | Nodes can be in multiple partitions |
| `nodes=N:ppn=P` (Torque) | `--nodes=N --ntasks-per-node=P` | Slurm separates tasks vs CPUs-per-task |
| `select=N:ncpus=C` (PBS Pro) | `--nodes=N --cpus-per-task=C` | Similar chunk concept |
| pbs_server | slurmctld | Central controller |
| pbs_mom | slurmd | Node daemon |
| `pbsnodes` | `sinfo` / `scontrol show node` | |
| Node properties | Node features | `--constraint` for selection |
| `qmgr` | `sacctmgr` + `scontrol` | Admin configuration |
| Fairshare tree | Fair Tree fairshare | Same concept, different implementation |
| Cray ALPS integration | Slurm native | Slurm runs natively on Cray |
| Job hooks | Job submit plugins (Lua) | `job_submit.lua` or `job_submit/pbs` plugin |

## Key Behavioral Differences

### 1. `cd $PBS_O_WORKDIR` Not Needed

PBS jobs start in `$HOME` by default, so PBS scripts commonly include `cd $PBS_O_WORKDIR`. Slurm starts in the submission directory by default, so this line can be removed.

### 2. Resource Specification Syntax

PBS uses `-l` for all resources in a single colon-separated string. Slurm uses individual `--` flags:

```bash
# PBS
#PBS -l nodes=2:ppn=16:mem=64gb:walltime=24:00:00

# Slurm (separate flags)
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=16
#SBATCH --mem=64G
#SBATCH --time=24:00:00
```

### 3. Memory Units

PBS uses `kb`, `mb`, `gb`. Slurm uses `K`, `M`, `G`, `T` (no `b`).

### 4. nodes:ppn vs ntasks vs cpus-per-task

This is the most common source of confusion:

- PBS `ppn` = processors per node (maps to either tasks or CPUs depending on use)
- For **MPI jobs:** `ppn=16` → `--ntasks-per-node=16`
- For **threaded jobs:** `ppn=16` → `--cpus-per-task=16 --ntasks=1`

### 5. PBS Pro `select` Chunks

PBS Pro's `select` syntax has no direct equivalent. Translate based on intent:

```bash
# PBS Pro: 4 chunks of 8 CPUs and 16GB each
#PBS -l select=4:ncpus=8:mem=16gb

# Slurm: 4 nodes, 8 CPUs each, 16GB each
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=8
#SBATCH --mem=16G
```

### 6. Slurm's job_submit/pbs Plugin

Slurm includes a `job_submit/pbs` plugin that can translate PBS directives in job scripts. This is useful during migration:

```ini
# slurm.conf
JobSubmitPlugins=job_submit/pbs
```

With this plugin, scripts containing `#PBS` directives are automatically translated to Slurm equivalents. This eases the transition but should not be relied on permanently.

=====
## References

- [SchedMD: Rosetta Stone of Workload Managers](https://slurm.schedmd.com/rosetta.html)
- [SchedMD: Job Submit Plugin (PBS)](https://slurm.schedmd.com/job_submit_plugins.html)
- [SchedMD: sbatch man page](https://slurm.schedmd.com/sbatch.html)