Exercise Validation

Every practical exercise in this training is periodically executed against a real ParallelCluster deployment and the captured output is reviewed for drift across Slurm, ParallelCluster, and AMI versions. This page summarizes the most recent run.

Last validation run

Field	Value
Run timestamp	2026-04-15T01:40:12Z
Cluster	slurm-training-val in us-east-2
Slurm version	25.11.4
ParallelCluster version	3.15.0
Head node AMI	ami-0e6c4d8fa29550a71
Head instance type	t3.medium
Harness Python	3.12.12
Terraform	1.14.5
AWS CLI	2.33.21
Training content SHA	4c70b94
Validation harness SHA	f378a41 (dirty)

Results

Status	Count
Passing	94
Failing	0
Skipped	2
Total	96

Per-exercise command output is captured on the exercise validation details page.

Modules with captures

Module	Exercises captured	Individual commands captured
admin/accounts-fairshare	4	13
admin/configuration	4	8
admin/high-availability	2	4
admin/installation	3	6
admin/maintenance-operations	2	7
admin/monitoring-accounting	4	5
admin/partitions-qos	4	7
admin/policies-priority	3	7
admin/resource-management	5	12
admin/slurm-architecture	2	5
admin/troubleshooting	5	12
deployment/aws-troubleshooting	2	5
user/best-practices	2	2
user/containers	2	4
user/environment-modules	3	7
user/getting-started	5	6
user/gpu-jobs	4	6
user/interactive-jobs	3	5
user/job-arrays	5	8
user/job-dependencies	4	5
user/managing-jobs	4	19
user/monitoring-jobs	5	5
user/parallel-mpi-jobs	4	4
user/recurring-jobs-scrontab	3	4
user/resource-requests	5	10
user/slurm-overview	2	4
user/submitting-jobs	5	7

Modules absent from this list fall into one of three categories:

• Conceptual-only (e.g. user/what-is-hpc-scheduling) — exercises are discussion prompts with no testable commands.
• Application-specific (e.g. applications/cryosparc-*) — require licensed software or dataset dependencies not on the validation cluster.
• Not yet reached — BATS test suites are written incrementally as exercises are reviewed. See the validation repo for the current coverage plan.

Lessons learned

Discoveries from validation runs that changed either the training content or the validation harness. Each entry is a short table row here with a link to the full writeup on its own page — detail lives there so this landing page stays scannable.

Date	Topic	Impact	Preview
2026-04-13	ParallelCluster /tmp is per-node; cross-node jobs can't see scratch files	both	The first end-to-end validation run of user/getting-started failed on two exercises that used mktemp -d -t to stage scratch files.
2026-04-13	Pcluster rejects --mem=<impossible> at submit time; resource-contention PENDING is on-prem-shaped	training	user/monitoring-jobs.md exercise 5 asks learners to request more memory than any node has so the job pends and squeue --start can report an estimated start time.
2026-04-13	Training hint in user/submitting-jobs uses /tmp/$USER — per-node on pcluster	training	user/submitting-jobs.md exercise 5 teaches --chdir by creating /tmp/$USER/test_chdir on the submitter host, then verifying the job ran there.
2026-04-13	AWS accounts start with EC2 G-instance quota = 0; GPU exercises blocked until quota raised	harness	The first full BATS run against the validation cluster failed all user/gpu-jobs tests with wait_for_job timing out.
2026-04-13	scrontab is disabled by default on ParallelCluster	training	user/recurring-jobs-scrontab.md teaches scrontab -e / scrontab -l / scrontab -r.
2026-04-13	Apptainer installed only on the head node; containerized sbatch jobs fail on compute	harness	user/containers.md exercise 2 submits a containerized batch job that the compute node has to execute.

About this process

The validation harness lives at bioteam/slurm-training-validation (private). It spins up an ephemeral ParallelCluster, installs the same software and accounting setup described in the training content, runs BATS test suites that invoke each exercise's commands, and writes structured JSONL records of every command, its output, and its exit code. Those records feed this page, the details page, and inline "Expected output" blocks inside each exercise's Hint / Solution toggle.

Cost of a full validation cycle is ~$3–5. Validation is not automated on CI — it's triggered manually by a maintainer when training content changes materially or a new Slurm / ParallelCluster version releases.