Skip to content

ParallelCluster /tmp is per-node; cross-node jobs can't see scratch files

Date: 2026-04-13. Impact: both.

← back to the Exercise Validation summary

What happened

The first end-to-end validation run of user/getting-started failed on two exercises that used mktemp -d -t to stage scratch files. The failure mode was subtle: the sbatch job reported COMPLETED and the test's read-back of the output file returned No such file or directory.

Root cause: on ParallelCluster (and most cloud HPC clusters), /tmp is a per-node directory. The head node and each compute node have their own separate /tmp. When the test creates a scratch directory under /tmp and submits a job whose working directory is that path, the job runs on a compute node and writes its output file to the compute node's /tmp. The test, still on the head node, looks in the head node's /tmp and sees nothing.

On traditional on-prem HPC this works by accident — many clusters mount /tmp on tmpfs or a shared filesystem, masking the problem. Cloud HPC surfaces the issue clearly.

Fix in the harness: scratch under $HOME (which ParallelCluster NFS-shares across head + compute by default) or under /shared (any explicit shared filesystem). Fix in the training content: user/best-practices.md and deployment/aws-parallelcluster.md now explicitly cover node-local vs shared scratch with worked examples; $SLURM_TMPDIR is the portable per-node scratch variable when per-node is actually what you want.

References

Fix commits

  • slurm-training-validation@0f2d0d5 — BATS setup uses $HOME, not /tmp, for per-test scratch
  • slurm-training@d2e6a39 — Ephemeral-storage coverage added to user/best-practices and both deployment modules