Skip to content

Apptainer installed only on the head node; containerized sbatch jobs fail on compute

Date: 2026-04-13. Impact: harness.

← back to the Exercise Validation summary

What happened

user/containers.md exercise 2 submits a containerized batch job that the compute node has to execute. The Ansible setup-software.yml playbook installs Apptainer via dnf (with an upstream EPEL-9 RPM fallback), but only targets the head node. pcluster compute nodes boot from a stock AL2023 AMI which does NOT ship Apptainer, so apptainer exec fails with command not found on any compute-bound job.

Four workable fix paths, in roughly increasing order of one-time setup effort:

  • Apptainer built from source into /shared/apptainer/ (EBS, visible on all nodes). Apptainer's ./mconfig --prefix=/shared/apptainer/ && make && make install produces a fully relocatable install. ~10-20 minutes of build time using a compute node's Go toolchain, then every compute node reaches apptainer at /shared/apptainer/bin/apptainer. Add the prefix to each job script's PATH (or ship an Lmod modulefile: /shared/modulefiles/apptainer/1.3.6.lua) and training users follow the same module load apptainer pattern they already know from Lmod. Most pcluster-idiomatic per the surrounding training content; also works on PCS with zero config change.

  • CustomActions.OnNodeConfigured runs a script at node boot. The script dnf install -y <apptainer-rpm-url>. Cleanest from pcluster's perspective but requires a cluster-config update and a cluster restart. MUST handle package-manager contention on freshly-booted nodes — cloud-init, dnf-automatic, unattended-upgrades can all be mid-operation. Recommended script skeleton:

#!/bin/bash
set -e
# Wait for any concurrent package-manager work to drain.
for i in $(seq 1 60); do
  fuser /var/cache/dnf/metadata_lock.pid \
        /var/lib/rpm/.rpm.lock \
        /var/lib/dnf/rpmdb_lock.pid 2>/dev/null || break
  echo "Waiting for dnf lock (${i}/60)" ; sleep 5
done
# Retry with backoff — DNS warmup and mirror flakes are common.
for attempt in 1 2 3; do
  dnf install -y \
    https://github.com/apptainer/apptainer/releases/download/v1.3.6/apptainer-1.3.6-1.x86_64.rpm \
    && exit 0
  sleep $(( attempt * 10 ))
done
exit 1

  • Custom AMI with Apptainer pre-baked, referenced via Image.CustomAmi in the pcluster config. Fastest compute- node boot time (no per-boot install step), most effort up-front (image-build pipeline).

  • RPM extracted + relocated into /shared/apptainer/. Fragile — apptainer's binaries have compiled-in paths for /usr/libexec/apptainer/, /etc/apptainer/, etc. Requires wrapper scripts with environment overrides for every invocation. Not recommended.

Until one of these lands, the containers ex2 test detects the missing binary on compute and skips with a clear message. Containers ex1 (the head-node apptainer pull) still runs.

Fix commits

  • slurm-training-validation@pending — containers.bats ex2 skips when apptainer is absent on compute