Getting Started

<!-- audience: L2 --> <!-- deployment: all --> <!-- sge-source: SGE "Getting work done" --> # Getting Started with Slurm ## Logging In Your first step is SSH-ing into the cluster's **login node** (also called a head node or submit node): ```bash ssh username@cluster.example.com ``` > **Important:** The login node is for writing scripts, submitting jobs, and light tasks. Do not run compute-intensive work directly on the login node -- that's what the scheduler is for. > **On-Prem Note:** You typically SSH to a dedicated login node provided by your system administrators (e.g., `ssh user@cluster.example.com`). > **ParallelCluster Note:** Connect via `pcluster ssh -n <cluster-name>` (which wraps SSH with the right key and IP), or SSH directly to the head node's public/private IP. > **PCS Note:** SSH to the managed login node(s) that PCS provisions for your cluster. Your administrator will provide the connection details. --- ## Checking the Cluster Before submitting jobs, get oriented: ```bash # What partitions are available? sinfo # Example output: PARTITION AVAIL TIMELIMIT NODES STATE NODELIST batch* up 7-00:00:00 48 idle node[001-048] gpu up 2-00:00:00 8 idle gpu[01-08] highmem up 7-00:00:00 4 idle bigmem[1-4] interactive up 4:00:00 48 idle node[001-048] ``` Key columns: - **PARTITION** -- name of the partition (`*` indicates the default) - **AVAIL** -- `up` means available for job submission - **TIMELIMIT** -- maximum walltime allowed - **NODES/STATE** -- how many nodes and their current status ```bash # What jobs are currently running? (all users) squeue # What are YOUR jobs? squeue --me ``` --- ## Your First Job Script A Slurm job script is a shell script with special `#SBATCH` directives that tell Slurm what resources you need. Create a file called `hello.sh`: ```bash #!/bin/bash #SBATCH --job-name=hello #SBATCH --output=hello_%j.out #SBATCH --time=00:05:00 #SBATCH --ntasks=1 #SBATCH --mem=1G echo "Hello from $(hostname)" echo "Job ID: $SLURM_JOB_ID" echo "Running on node: $SLURM_NODELIST" echo "Started at: $(date)" sleep 30 echo "Finished at: $(date)" ``` Let's break this down: | Directive | Meaning | |-----------|---------| | `#!/bin/bash` | Use bash as the shell interpreter | | `#SBATCH --job-name=hello` | Name your job (appears in `squeue`) | | `#SBATCH --output=hello_%j.out` | Where to write stdout (`%j` = job ID) | | `#SBATCH --time=00:05:00` | Maximum run time: 5 minutes | | `#SBATCH --ntasks=1` | Number of tasks (processes): 1 | | `#SBATCH --mem=1G` | Memory required: 1 GB | Everything after the `#SBATCH` lines is your normal shell script. --- ## Submitting the Job ```bash $ sbatch hello.sh Submitted batch job 12345 ``` Slurm returns a **job ID** (12345 in this example). You'll use this to track and manage your job. --- ## Checking Job Status ```bash $ squeue --me JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 12345 batch hello jdoe R 0:05 1 node042 ``` Key columns: - **JOBID** -- your job's unique identifier - **ST** (State) -- `R` = Running, `PD` = Pending, `CG` = Completing - **TIME** -- how long the job has been running - **NODELIST(REASON)** -- which node it's on, or why it's pending Common pending reasons: - `(Resources)` -- waiting for nodes/CPUs/memory to become available - `(Priority)` -- other higher-priority jobs are ahead of you - `(QOSMaxJobsPerUserLimit)` -- you've hit a per-user job limit --- ## Viewing Results Once the job completes, check the output file: ```bash $ cat hello_12345.out Hello from node042 Job ID: 12345 Running on node: node042 Started at: Sat Apr 12 14:30:01 EDT 2026 Finished at: Sat Apr 12 14:30:31 EDT 2026 ``` The default output file is `slurm-<jobid>.out`. We overrode this with `--output=hello_%j.out`. --- ## Key Environment Variables Inside your job script, Slurm sets several useful environment variables: | Variable | Description | Example Value | |----------|-------------|---------------| | `$SLURM_JOB_ID` | Job ID number | `12345` | | `$SLURM_JOB_NAME` | Job name | `hello` | | `$SLURM_NODELIST` | Assigned node(s) | `node042` | | `$SLURM_NTASKS` | Number of tasks | `1` | | `$SLURM_CPUS_PER_TASK` | CPUs per task | `1` | | `$SLURM_MEM_PER_NODE` | Memory per node (MB) | `1024` | | `$SLURM_SUBMIT_DIR` | Directory where `sbatch` was run | `/home/jdoe/project` | | `$SLURM_ARRAY_TASK_ID` | Array index (array jobs only) | `7` | These are useful for making your scripts dynamic and portable: ```bash #!/bin/bash #SBATCH --job-name=analysis #SBATCH --output=logs/%x_%j.out #SBATCH --time=01:00:00 echo "Job $SLURM_JOB_NAME (ID: $SLURM_JOB_ID)" echo "Running on: $SLURM_NODELIST" # Use submit directory for relative paths cd $SLURM_SUBMIT_DIR # Run your analysis ./run_analysis.sh --threads $SLURM_CPUS_PER_TASK ``` > **Tip:** `%x` in the output filename expands to the job name, `%j` to the job ID. This keeps your log files organized. --- ## Quick Workflow Summary ``` 1. Write your job script → vim my_job.sh 2. Submit it → sbatch my_job.sh 3. Check status → squeue --me 4. Wait for completion → (job runs on compute node) 5. Check results → cat slurm-<jobid>.out 6. Review accounting (optional) → sacct -j <jobid> ```

Exercises

1. Write and submit a simple job script

Create a file called first_job.sh that prints the hostname, the current date, and your username. Include #SBATCH directives for job name, output file, time limit (5 minutes), 1 task, and 1G of memory. Submit it with sbatch.

Hint / Solution

cat > first_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=first_job
#SBATCH --output=first_job_%j.out
#SBATCH --time=00:05:00
#SBATCH --ntasks=1
#SBATCH --mem=1G

echo "Hostname: $(hostname)"
echo "Date: $(date)"
echo "User: $USER"
EOF

sbatch first_job.sh

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch first_job.sh
Submitted batch job 42

1. Check your job status

After submitting the job above, use squeue to check whether your job is pending or running. Note the job ID, state, and which node it was assigned to.

Hint / Solution

squeue --me

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ squeue --me
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)

1. Find and read the output file

Once the job completes (it should finish quickly), find the output file and display its contents. The filename pattern includes the job ID.

Hint / Solution

# List matching output files
ls first_job_*.out

# Read the most recent one
cat first_job_<jobid>.out

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ ls gs3_*.out
gs3_42.out

$ cat gs3_42.out
Hello from batch-dy-compute-1

1. Check job accounting after completion

Use sacct to look up your completed job. Display the job ID, job name, elapsed time, state, and exit code.

Hint / Solution

sacct -j <jobid> --format=JobID,JobName,Elapsed,State,ExitCode

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sacct -j 42 --format=JobID,JobName,Elapsed,State,ExitCode
JobID           JobName    Elapsed      State ExitCode 
------------ ---------- ---------- ---------- -------- 
42             val_gs4   00:00:01  COMPLETED      0:0 
42.batch         batch   00:00:01  COMPLETED      0:0 

1. Explore the cluster

Use sinfo to determine how many partitions are available, which partition is the default, and what the maximum time limit is for each partition.

Hint / Solution

sinfo

The default partition is marked with `*` in the PARTITION column. The TIMELIMIT column shows the maximum walltime for each partition. Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      2  idle~ batch-dy-compute-[3-4]
batch*       up   infinite      2   idle batch-dy-compute-[1-2]
debug        up   infinite      1  idle% debug-dy-small-2
debug        up   infinite      1  idle~ debug-dy-small-1
gpu          up   infinite      1  idle~ gpu-dy-t4-1
highmem      up   infinite      1  idle~ highmem-dy-large-1

References

• SchedMD: sbatch man page
• SchedMD: squeue man page
• SchedMD: Quick Start User Guide
• SchedMD: Job State Codes