AWS accounts start with EC2 G-instance quota = 0; GPU exercises blocked until quota raised¶
Date: 2026-04-13. Impact: harness.
← back to the Exercise Validation summary
What happened¶
The first full BATS run against the validation cluster failed
all user/gpu-jobs tests with wait_for_job timing out. Root
cause was not Slurm, not the cluster config — it was the AWS
service quota. New accounts (and many mid-sized ones) have
Running On-Demand G and VT instances = 0 by default, so EC2
silently rejects every RunInstances call for g4dn, g5, g6,
etc. ParallelCluster's scale-up then fails and the node lands
in State=IDLE+CLOUD+POWERED_DOWN with
Reason=Scheduler health check failed.
For the single-GPU gpu-jobs.bats suite (g4dn.xlarge), requesting 4 vCPUs is sufficient. Recommended headroom 16-32 vCPUs to cover bigger g4dn types (up to g4dn.12xlarge = 48 vCPU). The request goes through AWS Service Quotas console (Service: EC2; Quota: "Running On-Demand G and VT instances").
The harness now pre-checks the gpu node's Slurm reason and skips gpu-jobs tests cleanly when the health check has failed, so a full validation run stays green on accounts without G-quota.
Resolved 2026-04-14: Sharedservices account G-instance quota raised to 16 vCPUs. Unconditional skip removed from gpu-jobs.bats; tests now gate only on partition presence and node health.
Fix commits¶
slurm-training-validation@pending— gpu-jobs.bats setup() skips when GPU node is unhealthyslurm-training-validation@pending— Remove unconditional GPU skip; quota now 16 vCPUs