Containers on Slurm

<!-- audience: L2, L3 --> <!-- deployment: all --> <!-- sge-source: not in SGE materials, modern topic --> # Containers on Slurm ## Why Containers in HPC? Containers package software and its dependencies into a portable unit. In HPC, containers solve common problems: - **Reproducibility:** Same container, same results, regardless of the host OS - **Dependency conflicts:** Different projects need different library versions - **Portability:** Move workloads between clusters, cloud, and laptops - **Complex software stacks:** Pre-built containers for tools like CryoSPARC, AlphaFold, TensorFlow --- ## Container Runtimes in HPC | Runtime | Use Case | Privilege Required | |---------|----------|--------------------| | **Apptainer/Singularity** | HPC standard, runs unprivileged | No | | **Docker** | Development, CI/CD | Yes (root, not available on most HPC clusters) | | **Podman** | Rootless Docker alternative | No | | **Slurm native OCI** | Direct `--container` support (newer) | No | **Apptainer** (formerly Singularity) is the de facto standard for HPC containers because it runs without root privileges and integrates naturally with shared filesystems and schedulers. --- ## Apptainer/Singularity with Slurm ### Running a Container in a Batch Job ```bash #!/bin/bash #SBATCH --job-name=container_job #SBATCH --time=02:00:00 #SBATCH --cpus-per-task=8 #SBATCH --mem=32G module load apptainer # or singularity apptainer exec my_image.sif ./run_analysis.sh ``` --- ## Common Apptainer Commands ```bash # Run a command inside a container apptainer exec image.sif <command> # Get an interactive shell apptainer shell image.sif # Run the container's default command apptainer run image.sif # Pull a container from Docker Hub apptainer pull docker://ubuntu:22.04 # Build from a definition file apptainer build my_image.sif my_definition.def ``` --- ## Bind Mounts (Accessing Your Data) By default, Apptainer mounts your home directory and current working directory. For other paths: ```bash # Bind external directories into the container apptainer exec --bind /data:/data,/scratch:/scratch image.sif ./analysis.sh # In a job script #!/bin/bash #SBATCH --job-name=container_analysis #SBATCH --time=04:00:00 apptainer exec \ --bind /shared/databases:/databases \ --bind $SLURM_SUBMIT_DIR:/work \ my_pipeline.sif \ /work/run_pipeline.sh ``` --- ## GPU Containers ```bash #!/bin/bash #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --time=12:00:00 module load apptainer # --nv flag enables NVIDIA GPU support apptainer exec --nv tensorflow_latest.sif python train.py ``` The `--nv` flag maps the host's NVIDIA drivers and GPU devices into the container. --- ## Real-World Container Examples ### AlphaFold via Container ```bash #!/bin/bash #SBATCH --job-name=alphafold #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --cpus-per-task=8 #SBATCH --mem=64G #SBATCH --time=24:00:00 module load apptainer apptainer exec --nv \ --bind /shared/alphafold_data:/data \ --bind $PWD:/output \ alphafold_2.3.sif \ python /app/run_alphafold.py \ --fasta_paths=/output/target.fasta \ --output_dir=/output/results \ --data_dir=/data ``` --- ## Bioconductor R Analysis ```bash #!/bin/bash #SBATCH --job-name=bioc_analysis #SBATCH --cpus-per-task=4 #SBATCH --mem=16G #SBATCH --time=02:00:00 module load apptainer apptainer exec \ --bind $SLURM_SUBMIT_DIR:/work \ bioconductor_3.18.sif \ Rscript /work/deseq2_analysis.R ``` --- ## Custom Pipeline Container ```bash #!/bin/bash #SBATCH --job-name=nf_rnaseq #SBATCH --cpus-per-task=16 #SBATCH --mem=64G #SBATCH --time=12:00:00 module load apptainer nextflow nextflow run nf-core/rnaseq \ -profile singularity \ --input samplesheet.csv \ --genome GRCh38 ``` Many pipeline frameworks (Nextflow, Snakemake) have built-in support for running each step in a container. --- ## Slurm Native OCI Container Support Since Slurm 21.08, there is native support for OCI containers via the `--container` flag: ```bash # Run a job using an OCI container bundle sbatch --container=/path/to/oci/bundle --time=01:00:00 --wrap="hostname" # Interactive srun --container=/path/to/oci/bundle --pty bash ``` This is a newer feature and requires administrator configuration (`oci.conf`). Most sites still use Apptainer/Singularity. > **ParallelCluster Note:** Since compute nodes are ephemeral, store Apptainer/Singularity `.sif` images on **shared storage** (FSx for Lustre or EFS), not on the head node's local disk. To avoid pull-time overhead, pre-stage images as part of a **custom AMI** or post-install script. > **PCS Note:** Similar to ParallelCluster, container images should live on shared storage attached to your PCS cluster. Custom AMIs can also be used to pre-bake images onto compute nodes. --- ## Pyxis and Enroot Pyxis is a Slurm (SPANK) plugin that runs jobs inside a container through Enroot, an NVIDIA user-namespace runtime. It adds options to `srun`/`sbatch`: ```bash srun --container-image=nvcr.io#nvidia/pytorch:24.09-py3 \ --container-mounts=/shared/$USER:/data \ python3 -c 'import torch; print(torch.__version__)' ``` - **ParallelCluster:** the AMIs ship Pyxis and Enroot. Confirm with `srun --help | grep container-image`. *(Not verified against the docs index here; check the ParallelCluster docs for your version.)* - Prefer it for **GPU and multi-node** jobs that pull Docker/NGC images directly; Enroot caches them as squashfs files, which you can keep on shared storage. - Prefer **Apptainer** for portable `.sif` files, on-prem sites without Pyxis, and workflow engines (Nextflow, Snakemake). - Do not run Pyxis and Slurm's native `--container` (`oci.conf`) together; SchedMD says to remove `oci.conf` at sites using a SPANK container plugin. - `--container-image` and `--container-mounts` come from the Pyxis project README; check `srun --help` for the options your cluster has. --- ## Ubuntu 24.04: Apptainer Fails with `uid_map` Symptom after moving nodes to Ubuntu 24.04 (Nextflow/nf-core steps fail at container launch): ``` FATAL: while creating user namespace: ... /proc/self/uid_map: Operation not permitted ``` Cause: `kernel.apparmor_restrict_unprivileged_userns=1` (the 24.04 default) blocks unprivileged user namespaces, which a non-setuid Apptainer needs. **Fix (admin):** a scoped AppArmor profile for the Apptainer starter only, in `/etc/apparmor.d/apptainer`: ``` abi <abi/4.0>, include <tunables/global> profile apptainer /opt/apptainer/libexec/apptainer/bin/starter{,-suid} flags=(unconfined) { userns, include if exists <local/apptainer> } ``` Match the path to your install prefix, then `sudo apparmor_parser -r -W /etc/apparmor.d/apptainer`. **Test on the real path** (a Slurm job or `ssh`), not an SSM session: the SSM agent runs in AppArmor complain mode and hides the failure. --- ## Tips 1. **Pull containers to shared storage**, not your home directory. Container images can be large (several GB). 2. **Use `--bind` for data paths.** Don't copy data into the container. 3. **Match GPU drivers.** The CUDA version inside your container must be compatible with the host's NVIDIA driver. The `--nv` flag handles this for most cases. 4. **Build on a machine with root access** (your laptop, a CI system), then copy the `.sif` file to the cluster. Building on the cluster itself requires `--fakeroot` support. 5. **Cache management.** Apptainer caches images in `$HOME/.apptainer`. Set `APPTAINER_CACHEDIR` to a scratch location if home is limited.

Exercises

1. Pull and inspect a container

Pull a container image from a public registry and examine its metadata. Use an image relevant to your work, or try the official Ubuntu image as a simple test. Inspect the image to see its labels, environment, and runscript.

Hint / Solution

# Load apptainer via Lmod (the canonical HPC pattern — the
# binary lives on a shared filesystem so the same install is
# reachable from head and compute nodes).
module load apptainer/1.5.4

# Pull an image (converts to SIF format). The ECR public mirror
# is more reliable than Docker Hub for automated runs; `--force`
# lets you re-pull over an existing ubuntu.sif.
apptainer pull --force ubuntu.sif docker://public.ecr.aws/docker/library/ubuntu:latest

# Inspect the image metadata
apptainer inspect ubuntu.sif

# See the image's default runscript
apptainer inspect --runscript ubuntu.sif

# See the image's environment setup
apptainer inspect --environment ubuntu.sif

# Quick test -- run a command inside the container
apptainer exec ubuntu.sif cat /etc/os-release

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ module load apptainer/1.5.4 && apptainer --version
apptainer version 1.5.4

$ module load apptainer/1.5.4 && apptainer pull --force ubuntu.sif docker://public.ecr.aws/docker/library/ubuntu:latest 2>&1
INFO:    Using cached SIF image

$ module load apptainer/1.5.4 && apptainer exec ubuntu.sif cat /etc/os-release
INFO:    squashfuse not found, will not be able to mount SIF or other squashfs files
INFO:    gocryptfs not found, will not be able to use gocryptfs
INFO:    Converting SIF file to temporary sandbox...
PRETTY_NAME="Ubuntu 26.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="26.04"
VERSION="26.04.1 LTS (Resolute Raccoon)"
VERSION_CODENAME=resolute
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=resolute
LOGO=ubuntu-logo
INFO:    Cleaning up image...

1. Run a containerized job

Write a Slurm batch script that runs a command inside a container. Use --bind to mount a data directory from the host into the container. Verify that the job can read input data and write output back to the host filesystem.

Hint / Solution

# Create test input data
mkdir -p /shared/$USER/container_test/input
mkdir -p /shared/$USER/container_test/output
echo "Hello from the host" > /shared/$USER/container_test/input/data.txt

# Write the job script — note the `module load apptainer` line:
# batch jobs run on compute nodes which don't have apptainer on
# PATH by default. Loading the module brings it onto PATH from
# the shared install.
cat > container_job.sh << 'EOF'
#!/bin/bash
#SBATCH --job-name=container_test
#SBATCH --output=container_%j.out
#SBATCH --time=00:10:00
#SBATCH --ntasks=1
#SBATCH --mem=2G

module load apptainer/1.5.4

DATADIR=/shared/$USER/container_test

apptainer exec \
  --bind $DATADIR/input:/mnt/input \
  --bind $DATADIR/output:/mnt/output \
  ubuntu.sif \
  bash -c "cat /mnt/input/data.txt && echo 'Processed at $(date)' > /mnt/output/result.txt"

echo "Job finished. Output:"
cat $DATADIR/output/result.txt
EOF

# Submit
sbatch container_job.sh

After the job completes, check `container_.out` for the output and verify that `/shared/$USER/container_test/output/result.txt` was written. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sbatch container_job.sh
Submitted batch job 42

References

• SchedMD: Containers Guide
• Apptainer Documentation
• SchedMD: OCI Container Support
• NVIDIA Pyxis
• Apptainer: AppArmor profile on Ubuntu 24.04 (issue #2027)