Containers on Slurm

<!-- audience: L2, L3 --> <!-- deployment: all --> <!-- sge-source: not in SGE materials, modern topic --> # Containers on Slurm ## Why Containers in HPC? Containers package software and its dependencies into a portable unit. In HPC, containers solve common problems: - **Reproducibility:** Same container, same results, regardless of the host OS - **Dependency conflicts:** Different projects need different library versions - **Portability:** Move workloads between clusters, cloud, and laptops - **Complex software stacks:** Pre-built containers for tools like CryoSPARC, AlphaFold, TensorFlow --- ## Container Runtimes in HPC | Runtime | Use Case | Privilege Required | |---------|----------|--------------------| | **Apptainer/Singularity** | HPC standard, runs unprivileged | No | | **Docker** | Development, CI/CD | Yes (root, not available on most HPC clusters) | | **Podman** | Rootless Docker alternative | No | | **Slurm native OCI** | Direct `--container` support (newer) | No | **Apptainer** (formerly Singularity) is the de facto standard for HPC containers because it runs without root privileges and integrates naturally with shared filesystems and schedulers. --- ## Apptainer/Singularity with Slurm ### Running a Container in a Batch Job ```bash #!/bin/bash #SBATCH --job-name=container_job #SBATCH --time=02:00:00 #SBATCH --cpus-per-task=8 #SBATCH --mem=32G module load apptainer # or singularity apptainer exec my_image.sif ./run_analysis.sh ``` --- ## Common Apptainer Commands ```bash # Run a command inside a container apptainer exec image.sif <command> # Get an interactive shell apptainer shell image.sif # Run the container's default command apptainer run image.sif # Pull a container from Docker Hub apptainer pull docker://ubuntu:22.04 # Build from a definition file apptainer build my_image.sif my_definition.def ``` --- ## Bind Mounts (Accessing Your Data) By default, Apptainer mounts your home directory and current working directory. For other paths: ```bash # Bind external directories into the container apptainer exec --bind /data:/data,/scratch:/scratch image.sif ./analysis.sh # In a job script #!/bin/bash #SBATCH --job-name=container_analysis #SBATCH --time=04:00:00 apptainer exec \ --bind /shared/databases:/databases \ --bind $SLURM_SUBMIT_DIR:/work \ my_pipeline.sif \ /work/run_pipeline.sh ``` --- ## GPU Containers ```bash #!/bin/bash #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --time=12:00:00 module load apptainer # --nv flag enables NVIDIA GPU support apptainer exec --nv tensorflow_latest.sif python train.py ``` The `--nv` flag maps the host's NVIDIA drivers and GPU devices into the container. --- ## Real-World Container Examples ### AlphaFold via Container ```bash #!/bin/bash #SBATCH --job-name=alphafold #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --cpus-per-task=8 #SBATCH --mem=64G #SBATCH --time=24:00:00 module load apptainer apptainer exec --nv \ --bind /shared/alphafold_data:/data \ --bind $PWD:/output \ alphafold_2.3.sif \ python /app/run_alphafold.py \ --fasta_paths=/output/target.fasta \ --output_dir=/output/results \ --data_dir=/data ``` --- ## Bioconductor R Analysis ```bash #!/bin/bash #SBATCH --job-name=bioc_analysis #SBATCH --cpus-per-task=4 #SBATCH --mem=16G #SBATCH --time=02:00:00 module load apptainer apptainer exec \ --bind $SLURM_SUBMIT_DIR:/work \ bioconductor_3.18.sif \ Rscript /work/deseq2_analysis.R ``` --- ## Custom Pipeline Container ```bash #!/bin/bash #SBATCH --job-name=nf_rnaseq #SBATCH --cpus-per-task=16 #SBATCH --mem=64G #SBATCH --time=12:00:00 module load apptainer nextflow nextflow run nf-core/rnaseq \ -profile singularity \ --input samplesheet.csv \ --genome GRCh38 ``` Many pipeline frameworks (Nextflow, Snakemake) have built-in support for running each step in a container. --- ## Slurm Native OCI Container Support Since Slurm 21.08, there is native support for OCI containers via the `--container` flag: ```bash # Run a job using an OCI container bundle sbatch --container=/path/to/oci/bundle --time=01:00:00 --wrap="hostname" # Interactive srun --container=/path/to/oci/bundle --pty bash ``` This is a newer feature and requires administrator configuration (`oci.conf`). Most sites still use Apptainer/Singularity. > **ParallelCluster Note:** Since compute nodes are ephemeral, store Apptainer/Singularity `.sif` images on **shared storage** (FSx for Lustre or EFS), not on the head node's local disk. To avoid pull-time overhead, pre-stage images as part of a **custom AMI** or post-install script. > **PCS Note:** Similar to ParallelCluster, container images should live on shared storage attached to your PCS cluster. Custom AMIs can also be used to pre-bake images onto compute nodes. --- ## Tips 1. **Pull containers to shared storage**, not your home directory. Container images can be large (several GB). 2. **Use `--bind` for data paths.** Don't copy data into the container. 3. **Match GPU drivers.** The CUDA version inside your container must be compatible with the host's NVIDIA driver. The `--nv` flag handles this for most cases. 4. **Build on a machine with root access** (your laptop, a CI system), then copy the `.sif` file to the cluster. Building on the cluster itself requires `--fakeroot` support. 5. **Cache management.** Apptainer caches images in `$HOME/.apptainer`. Set `APPTAINER_CACHEDIR` to a scratch location if home is limited.

Exercises

1. Pull and inspect a container

Pull a container image from a public registry and examine its metadata. Use an image relevant to your work, or try the official Ubuntu image as a simple test. Inspect the image to see its labels, environment, and runscript.

Hint / Solution

# Load apptainer via Lmod (the canonical HPC pattern — the
# binary lives on a shared filesystem so the same install is
# reachable from head and compute nodes).
module load apptainer/1.3.6

# Pull an image from Docker Hub (converts to SIF format)
apptainer pull ubuntu_latest.sif docker://ubuntu:latest

# Inspect the image metadata
apptainer inspect ubuntu_latest.sif

# See the image's default runscript
apptainer inspect --runscript ubuntu_latest.sif

# See the image's environment setup
apptainer inspect --environment ubuntu_latest.sif

# Quick test -- run a command inside the container
apptainer exec ubuntu_latest.sif cat /etc/os-release

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ module load apptainer/1.3.6 && apptainer --version
apptainer version 1.3.6

$ module load apptainer/1.3.6 && apptainer pull --force ubuntu.sif docker://public.ecr.aws/docker/library/ubuntu:latest 2>&1
INFO:    Using cached SIF image

$ module load apptainer/1.3.6 && apptainer exec ubuntu.sif cat /etc/os-release
INFO:    squashfuse not found, will not be able to mount SIF or other squashfs files
INFO:    gocryptfs not found, will not be able to use gocryptfs
INFO:    Converting SIF file to temporary sandbox...
PRETTY_NAME="Ubuntu 24.04.4 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04.4 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo
INFO:    Cleaning up image...

1. Run a containerized job

Write a Slurm batch script that runs a command inside a container. Use --bind to mount a data directory from the host into the container. Verify that the job can read input data and write output back to the host filesystem.

Hint / Solution

# Create test input data
mkdir -p /shared/$USER/container_test/input
mkdir -p /shared/$USER/container_test/output
echo "Hello from the host" > /shared/$USER/container_test/input/data.txt

# Write the job script — note the `module load apptainer` line:
# batch jobs run on compute nodes which don't have apptainer on
# PATH by default. Loading the module brings it onto PATH from
# the shared install.
cat > container_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=container_test
#SBATCH --output=container_%j.out
#SBATCH --time=00:10:00
#SBATCH --ntasks=1
#SBATCH --mem=2G

module load apptainer/1.3.6

DATADIR=/shared/$USER/container_test

apptainer exec \
  --bind $DATADIR/input:/mnt/input \
  --bind $DATADIR/output:/mnt/output \
  ubuntu_latest.sif \
  bash -c "cat /mnt/input/data.txt && echo 'Processed at $(date)' > /mnt/output/result.txt"

echo "Job finished. Output:"
cat $DATADIR/output/result.txt
EOF

# Submit
sbatch container_job.sh

After the job completes, check `container_.out` for the output and verify that `/shared/$USER/container_test/output/result.txt` was written. Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sbatch container_job.sh
Submitted batch job 42

References

• SchedMD: Containers Guide
• Apptainer Documentation
• SchedMD: OCI Container Support