Skip to content

Pcluster rejects --mem=<impossible> at submit time; resource-contention PENDING is on-prem-shaped

Date: 2026-04-13. Impact: training.

← back to the Exercise Validation summary

What happened

user/monitoring-jobs.md exercise 5 asks learners to request more memory than any node has so the job pends and squeue --start can report an estimated start time. The premise works on busy on-prem clusters but does not work cleanly on elastic ParallelCluster.

ParallelCluster declares every valid compute resource ahead of time. When a user submits --mem=200G to a partition whose largest declared node has 32 GB, slurmctld sees no satisfying node and sbatch rejects the submission at the CLI rather than queuing it as PENDING:

sbatch: error: Memory specification can not be satisfied
sbatch: error: Batch job submission failed: Requested node configuration is not available

Learners following the exercise verbatim on pcluster will hit a submission rejection, not the PENDING state the exercise is trying to teach. Two workable directions for the content:

  • Mention --hold as the portable way to force PENDING regardless of cluster elasticity (what the validation harness now uses).
  • Add a deployment callout to user/monitoring-jobs.md noting that resource-contention PENDING is transient or nonexistent on elastic clusters, and that --hold is the deterministic alternative for local experimentation.

The validation harness uses --hold so ex5 can run reliably across cluster types; the captured output for the exercise will therefore differ from the training hint's suggested command. That's acceptable for a teaching exercise where the point is what squeue --start shows, not how to force the pending state in the first place.

References

Fix commits

  • slurm-training-validation@ccee21c — monitoring-jobs.bats ex5 uses --hold to force PENDING deterministically