PBS to Slurm # PBS/Torque to Slurm Migration Guide This guide is for users and administrators transitioning from PBS Pro, OpenPBS, or Torque to Slurm. ## Command Mapping | Action | PBS/Torque | Slurm | Notes | |--------|-----------|-------|-------| | Submit batch job | `qsub script.sh` | `sbatch script.sh` | | | Delete job | `qdel job_id` | `scancel job_id` | | | Job status | `qstat` | `squeue` | | | My jobs | `qstat -u $USER` | `squeue --me` | | | Detailed job info | `qstat -f job_id` | `scontrol show job job_id` | | | Completed job info | `tracejob job_id` | `sacct -j job_id` | | | Queue info | `qstat -Q` | `sinfo` | | | Full queue info | `qstat -Qf` | `scontrol show partition` | | | Node status | `pbsnodes -l` | `sinfo -N -l` | | | Specific node | `pbsnodes node001` | `scontrol show node node001` | | | Interactive job | `qsub -I` | `srun --pty bash` | | | Interactive with X11 | `qsub -I -X` | `srun --pty --x11 bash` | | | Modify pending job | `qalter job_id` | `scontrol update JobId=job_id` | | | Hold job | `qhold job_id` | `scontrol hold job_id` | | | Release job | `qrls job_id` | `scontrol release job_id` | | | Move job to queue | `qmove queue job_id` | `scontrol update JobId=job_id Partition=part` | | | Server status | `qstat -Bf` | `scontrol show config` | | ## Job Script Directive Mapping | PBS Directive | Slurm Directive | Description | |--------------|-----------------|-------------| | `#PBS -N name` | `#SBATCH --job-name=name` | Job name | | `#PBS -o path` | `#SBATCH --output=path` | Stdout file | | `#PBS -e path` | `#SBATCH --error=path` | Stderr file | | `#PBS -j oe` | `#SBATCH --output=file` | Merge stdout/stderr (Slurm default) | | `#PBS -q queue` | `#SBATCH --partition=partition` | Queue/partition | | `#PBS -l walltime=HH:MM:SS` | `#SBATCH --time=HH:MM:SS` | Walltime | | `#PBS -l nodes=N:ppn=P` | `#SBATCH --nodes=N --ntasks-per-node=P` | Nodes and procs per node | | `#PBS -l select=N:ncpus=C:mem=M` | `#SBATCH --nodes=N --cpus-per-task=C --mem=M` | Resource chunk (PBS Pro) | | `#PBS -l mem=8gb` | `#SBATCH --mem=8G` | Total memory | | `#PBS -l pmem=4gb` | `#SBATCH --mem-per-cpu=4G` | Per-process memory | | `#PBS -l ngpus=1` | `#SBATCH --gres=gpu:1` | GPU request | | `#PBS -J 1-100` | `#SBATCH --array=1-100` | Job array (PBS Pro) | | `#PBS -t 1-100` | `#SBATCH --array=1-100` | Job array (Torque) | | `#PBS -A account` | `#SBATCH --account=account` | Account/project | | `#PBS -M email` | `#SBATCH --mail-user=email` | Email address | | `#PBS -m abe` | `#SBATCH --mail-type=BEGIN,END,FAIL` | Email events | | `#PBS -W depend=afterok:id` | `#SBATCH --dependency=afterok:id` | Dependency | | `#PBS -l feature=gpu` | `#SBATCH --constraint=gpu` | Node feature | | `#PBS -V` | `#SBATCH --export=ALL` | Export environment (Slurm default) | | `#PBS -d /path` | `#SBATCH --chdir=/path` | Working directory | | `#PBS -r y` | `#SBATCH --requeue` | Requeue on failure | ## Environment Variable Mapping | PBS Variable | Slurm Variable | Description | |-------------|----------------|-------------| | `$PBS_JOBID` | `$SLURM_JOB_ID` | Job ID | | `$PBS_JOBNAME` | `$SLURM_JOB_NAME` | Job name | | `$PBS_ARRAY_INDEX` | `$SLURM_ARRAY_TASK_ID` | Array task index | | `$PBS_NODEFILE` | `$SLURM_JOB_NODELIST` | Allocated nodes (format differs) | | `$PBS_NP` | `$SLURM_NTASKS` | Total processor count | | `$PBS_NUM_NODES` | `$SLURM_JOB_NUM_NODES` | Number of nodes | | `$PBS_NUM_PPN` | `$SLURM_NTASKS_PER_NODE` | Procs per node | | `$PBS_O_WORKDIR` | `$SLURM_SUBMIT_DIR` | Submission directory | | `$PBS_O_HOME` | `$HOME` | Home directory | | `$PBS_QUEUE` | `$SLURM_JOB_PARTITION` | Queue/partition | | `$TMPDIR` | `$TMPDIR` | Temp directory (site-dependent) | ## Job Script Translation: Side by Side ### PBS Version ```bash #!/bin/bash #PBS -N genome_align #PBS -o logs/align.out #PBS -e logs/align.err #PBS -q batch #PBS -l walltime=08:00:00 #PBS -l nodes=1:ppn=16 #PBS -l mem=32gb #PBS -V cd $PBS_O_WORKDIR module load bwa/0.7.17 samtools/1.17 bwa mem -t $PBS_NUM_PPN \ /data/reference/hg38.fa \ reads_R1.fastq.gz reads_R2.fastq.gz \ | samtools sort -@ 4 -o aligned.bam samtools index aligned.bam ``` ### Slurm Version ```bash #!/bin/bash #SBATCH --job-name=genome_align #SBATCH --output=logs/align.out #SBATCH --error=logs/align.err #SBATCH --partition=batch #SBATCH --time=08:00:00 #SBATCH --nodes=1 #SBATCH --cpus-per-task=16 #SBATCH --mem=32G module purge module load bwa/0.7.17 samtools/1.17 bwa mem -t $SLURM_CPUS_PER_TASK \ /data/reference/hg38.fa \ reads_R1.fastq.gz reads_R2.fastq.gz \ | samtools sort -@ 4 -o aligned.bam samtools index aligned.bam ``` ### Key Changes 1. `#PBS` becomes `#SBATCH` 2. `$PBS_O_WORKDIR` becomes `$SLURM_SUBMIT_DIR` (but `cd` is usually not needed -- Slurm starts in the submit dir by default) 3. `$PBS_NUM_PPN` becomes `$SLURM_CPUS_PER_TASK` 4. `-l nodes=1:ppn=16` becomes `--nodes=1 --cpus-per-task=16` (for threads) or `--ntasks-per-node=16` (for MPI) 5. `-l mem=32gb` becomes `--mem=32G` (note: `G` not `gb`) 6. `-V` is default in Slurm (environment is exported automatically) ### MPI Job Translation **PBS:** ```bash #!/bin/bash #PBS -N mpi_job #PBS -l walltime=24:00:00 #PBS -l nodes=4:ppn=32 #PBS -q batch cd $PBS_O_WORKDIR module load openmpi/4.1 mpirun -np $PBS_NP ./my_mpi_app ``` **Slurm:** ```bash #!/bin/bash #SBATCH --job-name=mpi_job #SBATCH --time=24:00:00 #SBATCH --nodes=4 #SBATCH --ntasks-per-node=32 #SBATCH --partition=batch module purge module load openmpi/4.1 srun ./my_mpi_app ``` Note: `srun` replaces `mpirun` and automatically knows the task count from the `#SBATCH` directives. ## Concept Mapping | PBS Concept | Slurm Equivalent | Key Differences | |-------------|------------------|-----------------| | Queue | Partition | Nodes can be in multiple partitions | | `nodes=N:ppn=P` (Torque) | `--nodes=N --ntasks-per-node=P` | Slurm separates tasks vs CPUs-per-task | | `select=N:ncpus=C` (PBS Pro) | `--nodes=N --cpus-per-task=C` | Similar chunk concept | | pbs_server | slurmctld | Central controller | | pbs_mom | slurmd | Node daemon | | `pbsnodes` | `sinfo` / `scontrol show node` | | | Node properties | Node features | `--constraint` for selection | | `qmgr` | `sacctmgr` + `scontrol` | Admin configuration | | Fairshare tree | Fair Tree fairshare | Same concept, different implementation | | Cray ALPS integration | Slurm native | Slurm runs natively on Cray | | Job hooks | Job submit plugins (Lua) | `job_submit.lua` or `job_submit/pbs` plugin | ## Key Behavioral Differences ### 1. `cd $PBS_O_WORKDIR` Not Needed PBS jobs start in `$HOME` by default, so PBS scripts commonly include `cd $PBS_O_WORKDIR`. Slurm starts in the submission directory by default, so this line can be removed. ### 2. Resource Specification Syntax PBS uses `-l` for all resources in a single colon-separated string. Slurm uses individual `--` flags: ```bash # PBS #PBS -l nodes=2:ppn=16:mem=64gb:walltime=24:00:00 # Slurm (separate flags) #SBATCH --nodes=2 #SBATCH --ntasks-per-node=16 #SBATCH --mem=64G #SBATCH --time=24:00:00 ``` ### 3. Memory Units PBS uses `kb`, `mb`, `gb`. Slurm uses `K`, `M`, `G`, `T` (no `b`). ### 4. nodes:ppn vs ntasks vs cpus-per-task This is the most common source of confusion: - PBS `ppn` = processors per node (maps to either tasks or CPUs depending on use) - For **MPI jobs:** `ppn=16` → `--ntasks-per-node=16` - For **threaded jobs:** `ppn=16` → `--cpus-per-task=16 --ntasks=1` ### 5. PBS Pro `select` Chunks PBS Pro's `select` syntax has no direct equivalent. Translate based on intent: ```bash # PBS Pro: 4 chunks of 8 CPUs and 16GB each #PBS -l select=4:ncpus=8:mem=16gb # Slurm: 4 nodes, 8 CPUs each, 16GB each #SBATCH --nodes=4 #SBATCH --ntasks-per-node=8 #SBATCH --mem=16G ``` ### 6. Slurm's job_submit/pbs Plugin Slurm includes a `job_submit/pbs` plugin that can translate PBS directives in job scripts. This is useful during migration: ```ini # slurm.conf JobSubmitPlugins=job_submit/pbs ``` With this plugin, scripts containing `#PBS` directives are automatically translated to Slurm equivalents. This eases the transition but should not be relied on permanently. ===== ## References - [SchedMD: Rosetta Stone of Workload Managers](https://slurm.schedmd.com/rosetta.html) - [SchedMD: Job Submit Plugin (PBS)](https://slurm.schedmd.com/job_submit_plugins.html) - [SchedMD: sbatch man page](https://slurm.schedmd.com/sbatch.html)