AWS ParallelCluster

<!-- audience: L4 --> <!-- deployment: aws-parallelcluster --> <!-- sge-source: none --> # AWS ParallelCluster ## Overview AWS ParallelCluster is an open-source cluster management tool that deploys and manages HPC clusters on AWS. It creates a fully configured Slurm cluster from a YAML configuration file, handling EC2 instances, networking, shared storage, and auto-scaling automatically. The latest release is **ParallelCluster 3.16.1** (September 2026, bundling Slurm 25.11.8). No ParallelCluster release ships Slurm 26.05 yet; [AWS PCS](aws-pcs.md) does. Notable 3.16 changes: Amazon Linux 2 and the AWS Batch scheduler are gone (Slurm is the only scheduler), a new `pcluster-diag` support-bundle tool, the head node exports NFSv4 only (lockd now on port 4045 -- open it if you firewall NFS), bootstrap files moved to `/opt/parallelcluster/tmp`, and the NVIDIA stack is driver 595 / CUDA 13.2 / DCGM 4.6. Version facts in this module are checked against the [document history](https://docs.aws.amazon.com/parallelcluster/latest/ug/document_history.html); the "Supported cluster and Slurm versions" page in the AWS docs is stale and stops at 3.13. Key differentiator from on-prem: **dynamic scaling**. ParallelCluster can scale compute nodes from zero to thousands based on pending jobs, then terminate idle instances to stop incurring costs. --- ## Architecture ``` ┌─────────────────────────────────────────────────────┐ │ AWS VPC │ │ │ │ ┌──────────────┐ ┌───────────────────────────┐ │ │ │ Head Node │ │ Compute Fleet │ │ │ │ (EC2) │ │ │ │ │ │ slurmctld │◄──►│ Static nodes (always on) │ │ │ │ slurmdbd │ │ Dynamic nodes (scale-to-0)│ │ │ │ (DB: RDS) │ │ │ │ │ │ sackd │ │ queue1-st-c7i-xl-[1-N] │ │ │ └──────┬───────┘ │ queue1-dy-c7i-xl-[1-M] │ │ │ │ └───────────────────────────┘ │ │ │ │ │ ┌──────▼───────┐ ┌───────────────────────────┐ │ │ │ Shared │ │ Login Node (optional) │ │ │ │ Storage │ │ User SSH access │ │ │ │ FSx / EFS │ └───────────────────────────┘ │ │ └──────────────┘ │ └─────────────────────────────────────────────────────┘ ``` ### What ParallelCluster Manages | Component | ParallelCluster Handles | You Configure | |-----------|------------------------|---------------| | **Slurm** | Installation, daemons, slurm.conf basics | CustomSlurmSettings, QOS, accounts | | **Compute nodes** | Launch, terminate, health checks | Instance types, MinCount/MaxCount | | **Networking** | VPC, subnets, security groups | Subnet IDs, placement groups | | **Storage** | Mount FSx/EFS/EBS | Storage type, size, throughput | | **Auth** | Munge setup | N/A (managed) | | **Scaling** | Resume/suspend programs | ScaledownIdletime | ### Login Nodes ParallelCluster 3.7.0+ supports managed login node groups. Login nodes use standard configless mode with slurmd to pull configuration from the head node. Users SSH to login nodes to submit and monitor jobs without accessing the head node directly. > **Note:** For multi-cluster scenarios (Slurm 25.05+), sackd can be manually configured on ParallelCluster login nodes to connect to multiple clusters from a single node. This is not a managed ParallelCluster feature -- you would need to create the sackd systemd services and manage the `SLURM_CONF` environment switching yourself. See the [on-premise configless](on-premise.md#multi-cluster-login-nodes-slurm-2505) section for the general sackd multi-cluster pattern. --- ## Cluster Configuration ParallelCluster uses a YAML configuration file. Here is a complete life science cluster example: ```yaml Region: us-east-2 Image: Os: alinux2023 HeadNode: InstanceType: c7i.2xlarge Networking: SubnetId: subnet-0abc123def456 Ssh: KeyName: my-hpc-key LocalStorage: RootVolume: Size: 100 Iam: AdditionalIamPolicies: - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore Scheduling: Scheduler: slurm SlurmSettings: ScaledownIdletime: 10 # Minutes before idle dynamic nodes terminate EnableMemoryBasedScheduling: Enabled: true SlurmQueues: # CPU queue -- genomics, bioinformatics - Name: cpu CapacityType: ONDEMAND ComputeResources: - Name: c7i-4xl Instances: - InstanceType: c7i.4xlarge # 16 vCPU, 32 GB MinCount: 0 # Scale to zero MaxCount: 100 - Name: c7i-16xl Instances: - InstanceType: c7i.16xlarge # 64 vCPU, 128 GB MinCount: 0 MaxCount: 20 Networking: SubnetIds: - subnet-0abc123def456 # High-memory queue -- genome assembly, large alignments - Name: highmem CapacityType: ONDEMAND ComputeResources: - Name: r7i-12xl Instances: - InstanceType: r7i.12xlarge # 48 vCPU, 384 GB MinCount: 0 MaxCount: 10 Networking: SubnetIds: - subnet-0abc123def456 # GPU queue -- cryo-EM, AlphaFold, ML training - Name: gpu CapacityType: ONDEMAND ComputeResources: - Name: g6-12xl Instances: - InstanceType: g6.12xlarge # 4x L4, 48 vCPU, 192 GB MinCount: 0 MaxCount: 8 - Name: p5-48xl Instances: - InstanceType: p5.48xlarge # 8x H100, 192 vCPU, 2 TB MinCount: 0 MaxCount: 4 Networking: SubnetIds: - subnet-0abc123def456 PlacementGroup: Enabled: true SharedStorage: - MountDir: /data Name: fsx-lustre StorageType: FsxLustre FsxLustreSettings: StorageCapacity: 7200 # GB (must be 1200 or increments of 2400) DeploymentType: PERSISTENT_2 PerUnitStorageThroughput: 250 # MB/s per TB - MountDir: /home Name: efs-home StorageType: Efs EfsSettings: ThroughputMode: elastic ``` --- ## Static vs. Dynamic Nodes This is the most important ParallelCluster concept for Slurm administrators. ### Static Nodes (always running) - Created at cluster launch, persist until cluster update or deletion - Named `<queue>-st-<resource>-<N>` (e.g., `cpu-st-c7i-4xl-1`) - Set with `MinCount` -- these instances are always running and billing - Use for: baseline capacity, always-on workloads ### Dynamic Nodes (scale-to-zero) - Created on demand when jobs are pending, terminated when idle - Named `<queue>-dy-<resource>-<N>` (e.g., `cpu-dy-c7i-4xl-1`) - Available count = `MaxCount - MinCount` - Terminated after `ScaledownIdletime` minutes of inactivity (default: 10) - Use for: burst capacity, cost optimization --- ## Node States and Scaling ### Node States in sinfo ```bash $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 20 idle~ cpu-dy-c7i-4xl-[1-20] cpu* up infinite 5 idle cpu-st-c7i-4xl-[1-5] gpu up infinite 8 idle~ gpu-dy-g6-12xl-[1-8] ``` | State Suffix | Meaning | |-------------|---------| | `idle` | Static node, running, no jobs | | `idle~` | Dynamic node, **not running** (powered down) | | `alloc` | Running a job | | `mix` | Partially allocated | | `idle#` | Dynamic node, **powering up** | | `down` | Node failed health check | The `~` suffix is critical -- it means the EC2 instance does not exist yet and costs nothing. ### Scaling Timeline 1. User submits job: `sbatch --partition=gpu myjob.sh` 2. slurmctld sees pending job, sets dynamic node to `idle#` (powering up) 3. ParallelCluster launches EC2 instance (~2-5 minutes for standard instances) 4. slurmd starts, node transitions to `alloc` 5. Job runs 6. Job completes, node becomes `idle` 7. After `ScaledownIdletime` (default 10 min) with no new jobs, node terminates 8. Node returns to `idle~` (powered down) --- ## Cloud Power-Save Keys On a cloud cluster, queue wait includes node boot time, and Slurm's power-save keys decide what a slow boot looks like. Read them before blaming the scheduler: ```bash scontrol show config | grep -E 'Suspend|Resume|SlurmctldParameters|SchedulerParameters' ``` | Key | ParallelCluster knob | What it looks like when wrong | |-----|---------------------|-------------------------------| | `ResumeTimeout` | None: deny-listed. Documented default 30 min (Slurm's stock default is 60 s) | Node misses it, goes DOWN, job is requeued. Reads as a job failure, not a slow boot; late boots show `Node unexpectedly rebooted.` | | `SuspendTime` | `SlurmSettings/ScaledownIdletime` (default 10 min); deny-listed in `CustomSlurmSettings` | Too short: idle nodes power off between bursts and every burst pays a cold start | | `SuspendTimeout` | `CustomSlurmSettings` (not deny-listed). Documented default 120 s | How fast a powered-down node is usable again, not how long it stays warm | | `SchedulerParameters=requeue_on_resume_failure` | `CustomSlurmSettings: - SchedulerParameters: ...` | Requeues batch jobs after a resume failure even if they asked for `--no-requeue` | | `SlurmctldParameters=rl_enable` | None: `SlurmctldParameters` is deny-listed | Clients are told to back off and sleep a second; reads as a slow scheduler during a submit flood | `requeue_on_resume_failure` is a `SchedulerParameters` value, not `SlurmctldParameters`. `SchedulerParameters` is one comma-separated list: start from the live value. Bootstrap failures that repeat trip protected mode instead: see [AWS Slurm Troubleshooting](aws-troubleshooting.md). --- ## Measuring and Shrinking Cold Start Cold start = `sbatch` submit to RUNNING on a node that was `idle~`. A warm node starts a job in about a second. Field range across three BioTeam engagements: about 4 min (recent AMI, minimal bootstrap) to about 20 min (2-year-old AMI, ansible-pull, ~130-package `apt install` on every boot). | # | Phase | Typical | Read it from | |---|-------|---------|--------------| | 1 | Slurm calls `ResumeProgram` (`CreateFleet`) | 5-15 s | `slurm_resume.log` | | 2 | EC2 instance pending to running | 3-30 s; longer on capacity errors | `slurm_resume.log`, CloudTrail | | 3 | Kernel boot, cloud-init | 30 s - 2 min | `cloud-init analyze blame` | | 4 | pcluster cookbook run + `OnNodeStart`/`OnNodeConfigured` | 1-15 min | `chef-client` CloudWatch stream, your bootstrap log | | 5 | `slurmd` registers with `slurmctld` | 10 s - 1 min | `slurmctld.log` | Phase 4 dominates. On one recent AMI the cookbook alone was about 4-4.5 min (fixed cost); everything above that is yours. --- ## Shrinking Cold Start: Bake, Don't Install > **ParallelCluster Note:** Never run `apt` or `dnf` in `OnNodeConfigured`. The root volume of a fresh node lazy-loads from its snapshot, so a package swap that takes 2-3 s on a warm host took about 2.5 min on a cold node (a 4 min 25 s Ansible play re-ran in 17 s warm). - **Bake into a custom AMI** (`Image/CustomAmi`), then guard the boot play so it is a no-op when the package is already there. - **`lmod` and `environment-modules` conflict** (mutual dpkg `Conflicts`): installing one removes the other, so a "keep both" approach is impossible. A second-stage image that re-runs the pcluster cookbook can reinstall `environment-modules` over a Lmod baked into the parent, so do the swap after the last cookbook stage. - **Time each phase**: `ANSIBLE_CALLBACKS_ENABLED=ansible.posix.profile_tasks` in the bootstrap, and on the node `grep -E 'Start-Date|End-Date|Commandline' /var/log/apt/history.log`. - **Run a canary** on a schedule and keep the numbers; an AMI bump that adds two minutes is otherwise invisible until users complain: ```bash #!/bin/bash # usage: coldstart-canary.sh <queue> <powered-down node, e.g. cpu-dy-c7i-4xl-1> q=${1:?queue}; n=${2:?node} [[ $(sinfo -h -n "$n" -o %T) == idle~ ]] || { echo "$n is not powered down" >&2; exit 1; } id=$(sbatch --parsable -p "$q" -w "$n" --chdir=/tmp --wrap='sleep 30') until [[ $(squeue -h -j "$id" -o %T) == RUNNING ]]; do sleep 5; done eval "$(scontrol show job "$id" -o | tr ' ' '\n' | grep -E '^(SubmitTime|StartTime)=')" echo "$(date -u +%FT%TZ),$q,$n,$(( $(date -d "$StartTime" +%s) - $(date -d "$SubmitTime" +%s) ))" ``` `--chdir=/tmp` matters when you run it from an SSM session; see the `0:53` cause in [AWS Slurm Troubleshooting](aws-troubleshooting.md). Run it three times per change; single runs vary by 30-60 s. --- ## Slurm Configuration Customization ParallelCluster generates `slurm.conf` automatically. You can customize it via `CustomSlurmSettings`, but some parameters are on a deny list. ### CustomSlurmSettings (cluster-level) ```yaml Scheduling: SlurmSettings: CustomSlurmSettings: - SchedulerParameters: bf_max_job_test=5000,bf_interval=30 - AccountingStorageEnforce: associations,limits,qos - PriorityType: priority/multifactor - PriorityWeightFairshare: 10000 - PriorityWeightAge: 1000 - PriorityWeightQOS: 5000 ``` ### CustomSlurmSettings (queue-level) ```yaml SlurmQueues: - Name: gpu CustomSlurmSettings: - MaxTime: 3-00:00:00 - DefaultTime: 04:00:00 - PreemptMode: REQUEUE ``` --- ## Deny-Listed Parameters ParallelCluster blocks certain slurm.conf parameters to prevent conflicts with its scaling logic: | Blocked Parameters | Reason | |-------------------|--------| | SlurmctldHost, SlurmUser | Managed by ParallelCluster | | ResumeProgram, SuspendProgram, ResumeFailProgram | Controls auto-scaling | | ResumeTimeout, SuspendTime | Scaling timing | | SlurmctldParameters, ReconfigFlags, TaskPlugin | Daemon behaviour owned by the cookbook | | GresTypes, Gres, State | GPU layout and `CLOUD` node state are generated (next section) | | Prolog, Epilog | Use ParallelCluster's prolog/epilog hooks | | Nodes, PartitionName | Derived from YAML config | | AccountingStorageType/Host/Port/User, JobAcctGatherType | Managed when `SlurmSettings/Database` is set | Use `CustomSlurmSettingsIncludeFile` for an external file (bypasses deny list validation, use with caution). --- ## How ParallelCluster Manages GPU GRES On bare metal you write `gres.conf`. On ParallelCluster you don't, and you can't: `GresTypes` (cluster level), `Gres` (compute-resource level) and `State` (queue level) are deny-listed in `CustomSlurmSettings` (since 3.6.0). ParallelCluster enumerates the GPUs from the instance type at cluster create and writes a three-level include chain (BioTeam capture from a 3.15.0 head node, `g5.xlarge`, queue `gpu`; the file names are generated, so check your release): ``` /opt/slurm/etc/gres.conf include slurm_parallelcluster_gres.conf /opt/slurm/etc/slurm_parallelcluster_gres.conf include /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf /opt/slurm/etc/pcluster/slurm_parallelcluster_gpu_gres.conf NodeName=gpu-dy-g5xlarge-[1-1] Name=gpu Type=a10g File=/dev/nvidia[0-0] ``` `slurm.conf` carries `GresTypes=gpu` and `Gres=gpu:<type>:<count>` on each `NodeName=`; the sibling `slurm_parallelcluster_<queue>_partition.conf` holds the `State=CLOUD` node and partition lines. All of it is regenerated on cluster updates: do not hand-edit. --- ## GPU GRES: No AutoDetect, No GPU Memory TRES `AutoDetect` is never set, and adding `AutoDetect=nvml` to a `State=CLOUD` node is rejected. `gres.conf(5)`: "AutoDetect cannot be used with cloud nodes." The failure is silent: ``` slurmctld.log: error: Cannot use AutoDetect on cloud/dynamic node "gpu-dy-g5xlarge-1" ``` `scontrol reconfigure` still exits 0 and `slurmctld` stays active. Tail `slurmctld.log` after any GRES edit. **Consequence:** `gres/gpumem` and `gres/gpuutil` have no NVML data source on cloud nodes and stay empty, whatever else you configure. Use DCGM/Prometheus or a prolog/epilog snapshot of `nvidia-smi` for per-job GPU memory or utilization. --- ## Tracking `gres/gpu` on ParallelCluster `gres/gpu` allocation accounting is achievable: `AccountingStorageTRES` is **not** deny-listed, and `gres/gpu` only needs to be appended to the defaults (`slurm.conf(5)`: the default TRES cannot be disabled, only appended to). Accounting itself needs `SlurmSettings/Database`: ```yaml Scheduling: SlurmSettings: Database: # external MySQL/RDS Uri: <db-endpoint>:3306 UserName: slurm PasswordSecretArn: <secret-arn> CustomSlurmSettings: - AccountingStorageTRES: gres/gpu ``` After this, `sacct -o JobID,AllocTRES` shows `gres/gpu=N`. See [GPU Jobs](../user/gpu-jobs.md) for the user view. --- ## Ephemeral Storage and `/scratch` The single storage topic that trips up HPC engineers moving from on-prem to AWS is what `/scratch` means. On most on-prem clusters, `/scratch` is a shared parallel filesystem (Lustre, GPFS, BeeGFS) that every node mounts — write on one node, read from any other. On ParallelCluster, **`/scratch` by default means local ephemeral NVMe on the compute node itself**. This section covers why, how to work with it, and when to replace or augment it with something shared. ### What ParallelCluster does by default When a compute instance type ships with local instance storage, pcluster (through its `cloudformation`-backed bootstrap) automatically partitions it, formats it ext4, and mounts it at `/scratch` on that node. If the instance has multiple local NVMe devices, pcluster stripes them into a RAID0 array for combined throughput. This is free — the NVMe is already part of the instance's hourly price — and it delivers 2–10 GB/s sequential throughput, which is an order of magnitude faster than any network filesystem pcluster can mount. Instance types that ship with local NVMe include: | Family | Typical use | Local NVMe available | |--------|------------|---------------------| | `g4dn.*`, `g5.*`, `g6.*` | GPU training / inference | 125 GB – 3.8 TB (SSD) | | `c5d.*`, `c6gd.*`, `c7gd.*` | Compute-optimized with storage | 50 GB – 1.9 TB | | `m5d.*`, `m6gd.*`, `m7gd.*` | General-purpose with storage | 75 GB – 3.6 TB | | `r5d.*`, `r6gd.*`, `r7gd.*` | Memory-optimized with storage | 75 GB – 3.6 TB | | `i3.*`, `i3en.*`, `i4i.*`, `i4g.*` | Storage-optimized, huge NVMe | 475 GB – 60 TB | Instance types **without** local storage (`t3.*`, `t4g.*`, the non-`d` variants of `c7i`/`m7i`/`r7i`, most Graviton unless suffixed `d`/`gd`) have no `/scratch` mounted by default at all — nothing is wired up because there's no local disk to mount. ### Why the ephemeral default is usually the right call Network filesystems on AWS — even FSx for Lustre — are slower for small-file or random-access workloads than a local NVMe attached to the same instance. For compute jobs that read the same data many times (particle images in cryo-EM reconstruction, features in ML training epochs, chunks of a sequence database in BLAST), staging to local NVMe once and hammering it for the rest of the job is the right pattern: ```bash #!/bin/bash #SBATCH --partition=gpu #SBATCH --gres=gpu:1 #SBATCH --time=04:00:00 # One-time stage-in from the shared filesystem rsync -a /fsx/project/data/ /scratch/data/ # Train; reads /scratch/data thousands of times at local NVMe speed python train.py --data /scratch/data --epochs 50 # Copy the model back before the node scales down cp checkpoints/model.pt /fsx/project/runs/$SLURM_JOB_ID/ ``` Three properties to internalize: 1. **It disappears with the node.** When Slurm's auto-scaling decides the compute node is idle and terminates it (default idle timeout is 10 minutes), every byte of `/scratch` goes away. This is usually what you want — it's why ephemeral is cheap — but it means nothing in `/scratch` should be considered durable. 2. **It's per-node.** Job submitted to 4 nodes sees 4 independent `/scratch` filesystems with nothing in common. An MPI job that wants coordinated checkpoint writes across ranks needs a shared filesystem, not `/scratch`. 3. **Size is tied to the instance type.** You pick `/scratch` capacity by choosing an instance family. A `g4dn.xlarge` comes with 125 GB of NVMe; `g4dn.2xlarge` has 225 GB; `i3.4xlarge` has 3.8 TB. There is no way to "resize" `/scratch` short of switching instance types. ### Overriding or replacing the default Use `ComputeSettings.LocalStorage.EphemeralVolume.MountDir` to put the ephemeral disk somewhere other than `/scratch` — for example, if your organization's training content or application configs expect a shared `/scratch` and you want to keep that convention on pcluster: ```yaml SlurmQueues: - Name: gpu ComputeResources: - Name: t4 InstanceType: g4dn.xlarge ComputeSettings: LocalStorage: EphemeralVolume: MountDir: /local-scratch # frees up /scratch for a shared mount ``` This lets you mount FSx for Lustre (or another shared filesystem) at `/scratch` and still use the free local NVMe via `/local-scratch`. To disable the automatic ephemeral mount entirely, set `MountDir` to an unused directory and ignore it, or omit instance types with instance storage from your queue definitions. ### Sizing strategies by workload | Workload pattern | Recommendation | |------------------|----------------| | Single-node jobs, heavy random I/O, data fits on local disk | Ephemeral `/scratch`; pick instance family by NVMe size | | Multi-node MPI with cross-rank I/O | FSx for Lustre (shared scratch); sized by concurrent throughput need | | Cryo-EM reconstruction (CryoSPARC) | FSx for Lustre for particles + ephemeral `/scratch` for SSD cache (classic split) | | Nextflow / Snakemake pipelines | FSx for Lustre or EFS mounted where your pipeline expects scratch | | Short validation or CI clusters | Single EBS mount at `/shared`; skip ephemeral and skip FSx — simplest, cheapest | | ML training with large dataset | FSx for Lustre for the dataset + per-GPU stage-in to `/scratch` per epoch | ### Portable code: `$SLURM_TMPDIR` When you want your job script to work the same on a pcluster with ephemeral `/scratch`, a pcluster without instance storage, and an on-prem cluster with shared `/scratch`, don't hard-code the path. Slurm sets `$SLURM_TMPDIR` to a per-job directory on every node; on pcluster/PCS it resolves to somewhere under `/scratch` (or a fallback path if the node has no instance storage), on on-prem clusters it's whatever your site's `slurm.conf` configures (`TmpFs=/scratch/%u/%j` is a common pattern). Referencing the variable rather than the literal path makes your scripts portable. ```bash # Portable: works on any cluster that sets SLURM_TMPDIR cp /shared/input.bam $SLURM_TMPDIR/ cd $SLURM_TMPDIR samtools sort input.bam -o sorted.bam cp sorted.bam /shared/output/ ``` ### Example: validation cluster config The `bioteam/slurm-training-validation` repo runs a minimal cluster for testing exercise commands. It deliberately uses *only* EBS `/shared` and accepts pcluster's default per-node ephemeral `/scratch` (on the GPU queue only, since t3-based queues have no instance storage): ```yaml SharedStorage: - MountDir: /shared Name: shared StorageType: Ebs EbsSettings: { VolumeType: gp3, Size: 30 } # No /scratch in SharedStorage — let pcluster handle it per-node ``` That's the "simplest thing that works" pattern for short-lived test clusters. Production clusters add FSx for Lustre (or ONTAP / OpenZFS depending on access pattern needs) to the same config. --- ## Storage Options ParallelCluster supports multiple shared storage types: Amazon EBS, Amazon EFS, FSx for Lustre, FSx for ONTAP, and FSx for OpenZFS. ### FSx for Lustre Best for: high-throughput parallel I/O (cryo-EM particle data, large genomics datasets) ```yaml SharedStorage: - MountDir: /scratch StorageType: FsxLustre FsxLustreSettings: StorageCapacity: 7200 DeploymentType: PERSISTENT_2 # Durable, multi-AZ PerUnitStorageThroughput: 250 # MB/s per TB ``` - PERSISTENT_2: data survives filesystem deletion, replicated - SCRATCH_2: higher throughput, data lost if filesystem deleted (temp workloads) - Can link to S3 for data import/export - **EFA-enabled FSx Lustre** (ParallelCluster 3.13+): enables higher throughput via Elastic Fabric Adapter for storage traffic ### EFS (Elastic File System) Best for: home directories, software installations, shared config ```yaml SharedStorage: - MountDir: /home StorageType: Efs EfsSettings: ThroughputMode: elastic # Scales automatically ``` - NFS-compatible, fully managed - Lower throughput than FSx Lustre - Good for small-file workloads --- ## Storage Options (continued) ### FSx for NetApp ONTAP Best for: multi-protocol access (NFS + SMB), data tiering, snapshots ```yaml SharedStorage: - MountDir: /data StorageType: FsxOntap FsxOntapSettings: VolumeId: fsvol-0abc123def456 ``` - Multi-protocol (NFS, SMB, iSCSI) - Built-in data tiering to S3 - Useful when researchers need both Linux and Windows access ### EBS (Elastic Block Store) Best for: head node local storage only ```yaml HeadNode: LocalStorage: RootVolume: Size: 200 VolumeType: gp3 ``` - Attached to a single instance - Not shared -- only use for head node or as local scratch on compute nodes --- ## Networking ### EFA (Elastic Fabric Adapter) For MPI-heavy workloads (molecular dynamics, quantum chemistry), enable EFA: ```yaml SlurmQueues: - Name: mpi ComputeResources: - Name: hpc6a-48xl Instances: - InstanceType: hpc6a.48xlarge # EFA-enabled HPC instance MinCount: 0 MaxCount: 16 Efa: Enabled: true Networking: SubnetIds: - subnet-0abc123def456 PlacementGroup: Enabled: true # Co-locate instances for low latency ``` ### Placement Groups Enable for latency-sensitive workloads: ```yaml Networking: PlacementGroup: Enabled: true # Cluster placement group ``` Placement groups ensure instances launch in close physical proximity, reducing network latency. Essential for multi-node MPI jobs. --- ## Instance Types for Life Science | Workload | Recommended Instance | vCPU | Memory | GPU | Notes | |----------|---------------------|------|--------|-----|-------| | Genomics (BLAST, BWA) | c7i.4xlarge - c7i.16xlarge | 16-64 | 32-128 GB | - | CPU-bound | | Genome assembly | r7i.12xlarge - r7i.24xlarge | 48-96 | 384-768 GB | - | Memory-bound | | Cryo-EM (RELION) | g6e.12xlarge | 48 | 384 GB | 4x L40S | GPU-accelerated | | Cryo-EM (CryoSPARC) | p5.48xlarge | 192 | 2 TB | 8x H100 | Multi-GPU | | AlphaFold | g6e.xlarge - g6e.12xlarge | 4-48 | 32-384 GB | 1-4x L40S | GPU-bound | | ML training | p5.48xlarge | 192 | 2 TB | 8x H100 | Large-scale GPU | | Molecular dynamics | hpc6a.48xlarge | 96 | 384 GB | - | EFA for MPI | Instance generations and sizes change; the table is a starting point, not a sizing. On-Demand prices (Linux, us-east-2, AWS price list of 2026-10-15): c7i.4xlarge $0.714/h, r7i.12xlarge $3.175/h, g6.12xlarge $4.602/h, g6e.12xlarge $10.493/h, p5.48xlarge $55.04/h. Re-check before quoting; [Cost Allocation](../leadership/cost-allocation.md) shows the per-GPU-hour arithmetic. --- ## Cost Management ### Scale-to-Zero Pattern The most impactful cost optimization: set `MinCount: 0` for all queues. ```yaml ComputeResources: - Name: c7i-4xl MinCount: 0 # No instances running when idle MaxCount: 100 # Can burst to 100 ``` Users experience a 2-5 minute startup delay for the first job, but pay nothing when the cluster is idle. --- ## Spot Instances and Idle Timeout ### Spot Instances For fault-tolerant workloads (job arrays, parameter sweeps), use Spot: ```yaml SlurmQueues: - Name: spot-cpu CapacityType: SPOT AllocationStrategy: capacity-optimized ComputeResources: - Name: spot-compute Instances: - InstanceType: c7i.4xlarge - InstanceType: c7a.4xlarge - InstanceType: c6i.4xlarge # Multiple types for availability MinCount: 0 MaxCount: 200 ``` - Spot can save 60-90% vs. On-Demand - Instances can be reclaimed with 2-minute warning - Use with `--requeue` so interrupted jobs restart automatically ### ScaledownIdletime Balance responsiveness vs. cost: ```yaml SlurmSettings: ScaledownIdletime: 10 # Default: terminate after 10 min idle ``` | Value | Trade-off | |-------|-----------| | 2-5 min | Aggressive savings, but frequent launch delays for bursty users | | 10 min (default) | Good balance for most life science workloads | | 30-60 min | Keep warm for interactive users, higher cost | --- ## Cluster Lifecycle ### Create ```bash # Install the CLI pip install aws-parallelcluster # Create cluster from config pcluster create-cluster --cluster-name biocluster \ --cluster-configuration cluster-config.yaml # Monitor creation pcluster describe-cluster --cluster-name biocluster ``` > **ParallelCluster Note (time-boxed: remove when 3.16 is the floor):** On 3.15.0 only, `create-cluster` with Slurm accounting fails at `HeadNodeWaitCondition` if the cluster name has uppercase letters: `sacctmgr` lowercases cluster names and the 3.15.0 bootstrap check is case-sensitive. Use an all-lowercase name. 3.16.0 fixed it (release notes: "Fix cluster creation failure caused by Slurm accounting bootstrap failing when ClusterName is overridden via custom Slurm settings or when the cluster name contains uppercase letters"). 3.16.0 also caps cluster names at 40 characters with `Database` or `ExternalSlurmdbd`. Separately, Slurm 25.x `slurmdbd` needs MySQL 8.0.4 or newer (BioTeam field note: `REGEXP_REPLACE` missing on an RDS still at 5.7, so the cluster never comes up whatever the name). --- ## Cluster Updates and Access ### Update ```bash # Update config (some changes require compute fleet stop) pcluster update-cluster --cluster-name biocluster \ --cluster-configuration updated-config.yaml # For changes that require fleet stop: pcluster update-compute-fleet --cluster-name biocluster --status STOP_REQUESTED # ... then update, then: pcluster update-compute-fleet --cluster-name biocluster --status START_REQUESTED ``` Changes that do NOT require fleet stop: - Adding new queues or compute resources - Increasing MaxCount - Increasing MinCount (with matching MaxCount increase) - Updating CustomSlurmSettings ### SSH Access ```bash pcluster ssh --cluster-name biocluster -i ~/my-hpc-key.pem ``` ### Delete ```bash pcluster delete-cluster --cluster-name biocluster ``` --- ## Post-Creation Slurm Setup After cluster creation, configure Slurm accounting (not managed by ParallelCluster YAML): ```bash # SSH to head node pcluster ssh --cluster-name biocluster # Create accounts and users sudo sacctmgr add account research Description="Research" sudo sacctmgr add account cryoem parent=research Description="Cryo-EM" sudo sacctmgr add user jsmith Account=research # Create QOS sudo sacctmgr add qos gpu_priority Priority=100 MaxTRESPerUser=gpu=8 sudo sacctmgr add qos burst MaxSubmitJobsPerUser=500 ``` ### Why Accounting Matters on ParallelCluster > **Best Practice:** Accounting is not on by default. A stock ParallelCluster has no `slurmdbd` (`AccountingStorageType` is unset and `sacct` reports that accounting storage is disabled) until you add a `SlurmSettings/Database` block pointing at an external MySQL/RDS database. Once it is on, accounting is only useful when you create accounts, users, and QOS entries (as shown above). Always complete this setup. Without accounting fully configured, the following capabilities are **unavailable**: > > - **Fairshare scheduling** (priority/multifactor) -- cannot weight user/group priority by historical usage > - **QOS enforcement** -- cannot set resource limits per user, account, or group > - **Job history and reporting** (`sacct`, `sreport`) -- no post-job resource usage data > - **License-aware scheduling** -- cannot integrate FLEXlm license counts (required for Schrodinger and other commercial software) > - **Account-based usage tracking and chargeback** -- no way to attribute compute hours to grants or departments > - **Resource limit enforcement** (MaxJobs, MaxSubmitJobs, GrpTRES) -- cannot prevent any single user from consuming the entire cluster > > On cloud clusters where every compute hour has a direct dollar cost, accounting is arguably more important than on-prem. Without it, you have no mechanism to enforce spend limits per group, no fairshare to balance usage across teams, and no historical data for cost allocation. > **ParallelCluster Note:** When `SlurmSettings/Database` is configured, ParallelCluster owns `AccountingStorageType`, `AccountingStorageHost`, `AccountingStoragePort`, `AccountingStorageUser` and `JobAcctGatherType` (deny-listed in `CustomSlurmSettings`). You do not set these -- create accounts, users, and QOS via `sacctmgr`. To track GPUs in `sacct`, see [GPU GRES](#how-parallelcluster-manages-gpu-gres). --- ## Key Differences from On-Prem | Aspect | On-Prem | ParallelCluster | |--------|---------|-----------------| | **Slurm version** | You choose | Bundled with PC version | | **slurm.conf** | Full control | Generated + CustomSlurmSettings | | **HA** | slurmctld primary/backup | Single head node (no native HA) | | **Node scaling** | Fixed | Dynamic (scale-to-zero) | | **Node names** | You define | Auto-generated (`queue-st/dy-resource-N`) | | **GRES/GPU** | Manual gres.conf | Pre-enumerated per instance type at cluster create (explicit `File=` entries; no AutoDetect) | | **Storage** | Your infrastructure | FSx Lustre, EFS, EBS | | **Networking** | IB, Ethernet | EFA, VPC | | **Prolog/Epilog** | Direct in slurm.conf | ParallelCluster hooks (deny-listed) | | **OS** | Any Linux | Amazon Linux 2023, Ubuntu 22.04/24.04, RHEL 8/9, Rocky 8/9 (Rocky needs a custom AMI) | | **Upgrades** | In-place | New cluster (blue-green) | --- ## Troubleshooting ### Nodes stuck in `idle#` (powering up) ```bash # Check cloud node status scontrol show node cpu-dy-c7i-4xl-1 # Check ParallelCluster logs sudo tail -f /var/log/parallelcluster/clustermgtd.log sudo tail -f /var/log/parallelcluster/slurm_resume.log ``` Common causes: insufficient capacity in the AZ, service quotas exceeded, subnet out of IPs. ### Nodes going to `down` state ```bash # Check why scontrol show node cpu-dy-c7i-4xl-1 | grep Reason # ParallelCluster health check logs sudo tail -f /var/log/parallelcluster/computemgtd.log ``` ### Jobs pending with `(ReqNodeNotAvail)` Normal for dynamic nodes -- the node is launching. Wait 2-5 minutes. If persistent: ```bash # Check if it's a capacity issue squeue -j JOBID -o "%i %j %T %r" # Check EC2 service quotas aws service-quotas get-service-quota \ --service-code ec2 \ --quota-code L-1216C47A # Running On-Demand Standard instances ```

Related Modules

• On-Premise Deployment -- traditional deployment for comparison
• AWS PCS -- AWS managed alternative
• GPU Jobs -- user guide for GPU workloads on the cluster
• Accounts & Fairshare -- post-creation accounting setup
• Cost Allocation -- chargeback with cloud cost attribution

References

• AWS ParallelCluster Documentation
• AWS ParallelCluster Slurm Workload Manager
• Slurm Configuration Customization
• Cluster Capacity Size and Update
• Multiple Instance Type Allocation
• SchedMD: Slurm Power Saving Guide
• AWS ParallelCluster Release Notes and Document History (3.16.0 uppercase-cluster-name fix, 3.6.0 CustomSlurmSettings)
• AWS ParallelCluster: Slurm Accounting
• AWS ParallelCluster: Slurm Multiple Queue Mode Guide (ResumeTimeout, SuspendTimeout defaults)
• AWS ParallelCluster: Scheduling Configuration (Database, ScaledownIdletime)
• AWS ParallelCluster: Protected Mode
• SchedMD: gres.conf
• SchedMD: slurm.conf
• Amazon EC2 On-Demand Pricing