Interactive Jobs

# Interactive Jobs

## Why Interactive Jobs?

Sometimes you need to work directly on a compute node -- to test code, debug a pipeline, visualize data, or run software that requires human input. Interactive jobs let you do this through the scheduler, so your work is tracked and other users' jobs aren't affected.

> **Rule:** Never run compute-intensive work on the login node. Use interactive jobs instead.

---

## srun: Run a Command Interactively

`srun` allocates resources and runs a command immediately:

### Run a Single Command

```bash
# Run hostname on a compute node
$ srun hostname
node042

# Run a quick test on a GPU node
$ srun --partition=gpu --gres=gpu:1 --time=00:30:00 nvidia-smi
```

### Get an Interactive Shell

```bash
$ srun --pty --time=02:00:00 --mem=8G bash
[jdoe@node042 ~]$

# Now you're on a compute node -- run commands interactively
[jdoe@node042 ~]$ module load python/3.11
[jdoe@node042 ~]$ python my_script.py
[jdoe@node042 ~]$ exit   # Release the allocation
```

The `--pty` flag gives you a proper terminal (required for interactive shells).

### Interactive GPU Session

```bash
$ srun --pty --partition=gpu --gres=gpu:1 --cpus-per-task=4 --mem=32G --time=04:00:00 bash
[jdoe@gpu03 ~]$ nvidia-smi
[jdoe@gpu03 ~]$ python train_model.py    # Test your GPU code
[jdoe@gpu03 ~]$ exit
```

---

## salloc: Allocate Resources, Then Use Them

`salloc` obtains a resource allocation without immediately running anything. You then use `srun` within that allocation to launch work:

```bash
# Request resources
$ salloc --time=04:00:00 --cpus-per-task=8 --mem=32G
salloc: Granted job allocation 12355
salloc: Nodes node042 are ready for job

# You're still on the login node, but have reserved compute resources
$ srun hostname
node042

# Run multiple commands on your allocated resources
$ srun ./step1.sh
$ srun ./step2.sh

# SSH to the allocated node directly (if permitted)
$ ssh node042

# Release the allocation when done
$ exit
salloc: Relinquishing job allocation 12355
```

### srun vs. salloc: When to Use Which

| Use Case | Command |
|----------|---------|
| Quick single command on a compute node | `srun <command>` |
| Interactive shell session | `srun --pty bash` |
| Multiple sequential commands on reserved resources | `salloc` then `srun` |
| Development/debugging session | Either works; `salloc` is more flexible |
| GUI application (X11) | `srun --x11 <app>` or `salloc --x11` |

---

## X11 Forwarding

To run graphical applications on compute nodes (e.g., Maestro, PyMOL, RStudio):

```bash
# First: SSH to the login node with X11 forwarding
$ ssh -X username@cluster.example.com

# Then request an interactive session with X11
$ srun --pty --x11 --time=02:00:00 --mem=8G bash
[jdoe@node042 ~]$ module load pymol
[jdoe@node042 ~]$ pymol    # Opens GUI on your local display
```

Or with `salloc`:

```bash
$ salloc --x11 --time=02:00:00 --mem=8G
$ srun --x11 pymol
```

> **Note:** X11 forwarding requires your SSH connection to support it (`ssh -X` or `ssh -Y`). Performance over high-latency connections can be poor -- consider VNC or a web-based tool instead.

---

## Jupyter Notebooks on Compute Nodes

A common modern workflow for interactive data science:

### Method 1: SSH Tunnel

```bash
# Step 1: Start Jupyter on a compute node
$ srun --pty --time=04:00:00 --cpus-per-task=4 --mem=16G bash
[jdoe@node042 ~]$ module load anaconda
[jdoe@node042 ~]$ jupyter notebook --no-browser --port=8888 --ip=0.0.0.0

# Step 2: From your local machine, create an SSH tunnel
$ ssh -L 8888:node042:8888 username@cluster.example.com

# Step 3: Open http://localhost:8888 in your browser
```

### Method 2: Job Script with Port Forwarding Info

```bash
#!/bin/bash
#SBATCH --job-name=jupyter
#SBATCH --output=jupyter_%j.log
#SBATCH --time=08:00:00
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G

module load anaconda

NODE=$(hostname -s)
PORT=8888

echo "============================================"
echo "Jupyter is running on: $NODE:$PORT"
echo "SSH tunnel command:"
echo "  ssh -L $PORT:$NODE:$PORT $USER@cluster.example.com"
echo "Then open: http://localhost:$PORT"
echo "============================================"

jupyter notebook --no-browser --port=$PORT --ip=0.0.0.0
```

---

## Timeouts and Idle Sessions

Interactive jobs consume resources even when you're idle. Be mindful:

- Set reasonable time limits (`--time`)
- Exit when you're done (`exit` or `Ctrl-D`)
- On some clusters, idle interactive sessions are automatically killed
- If your SSH connection drops, the interactive job may continue running until its time limit

> **ParallelCluster Note:** Interactive jobs on dynamic nodes may take longer to start because the node needs to launch first. Consider using static nodes (if available) or the `interactive` partition for quicker startup.

---

## Quick Reference

| Action | Command |
|--------|---------|
| Interactive shell | `srun --pty --time=HH:MM:SS bash` |
| Interactive shell with GPU | `srun --pty --partition=gpu --gres=gpu:1 --time=HH:MM:SS bash` |
| Single command on compute node | `srun --time=00:10:00 <command>` |
| Reserve resources for later use | `salloc --time=HH:MM:SS` |
| X11 graphical session | `srun --pty --x11 --time=HH:MM:SS bash` |
| Release salloc allocation | `exit` |

Exercises¶

Get an interactive shell with srun

Use srun to start an interactive bash session on a compute node with 2 CPUs, 4G of memory, and a 30-minute time limit. Once on the node, confirm you are not on the login node by running hostname, then exit cleanly.

Hint / Solution

srun --pty --cpus-per-task=2 --mem=4G --time=00:30:00 bash

# On the compute node:
hostname       # Should show a compute node name, not the login node
nproc          # Should show at least 2
exit           # Release the allocation

Allocate resources with salloc, then run commands

Use salloc to reserve 4 CPUs and 8G of memory for 1 hour. Then use srun within that allocation to run hostname and free -m on the allocated node. Release the allocation when done.

Hint / Solution

salloc --cpus-per-task=4 --mem=8G --time=01:00:00

# You're still on the login node but have reserved resources
srun hostname
srun free -m

# Release
exit

Request X11 forwarding for a graphical application

First, SSH to the cluster with X11 forwarding enabled. Then use srun with --x11 to start an interactive session. Verify X11 is working by running xeyes or xclock (if available), or by checking that the $DISPLAY environment variable is set.

Hint / Solution

# From your local machine:
ssh -X username@cluster.example.com

# On the login node:
srun --pty --x11 --time=00:30:00 --mem=4G bash

# On the compute node, verify X11:
echo $DISPLAY       # Should be set (e.g., localhost:10.0)
xclock &            # Should open a clock window on your display
exit

Run a single command on a compute node without an interactive shell

Use srun (without --pty or bash) to run a single command on a compute node. Check which Linux kernel version is running on the compute nodes by executing uname -a.

Hint / Solution

srun --time=00:05:00 uname -a

This runs the command on a compute node and prints the result directly to your terminal without starting an interactive session.

Interactive Jobs

Exercises¶

References¶