AWS ParallelCluster

<!-- audience: L4 --> <!-- deployment: aws-parallelcluster --> <!-- sge-source: none --> # AWS ParallelCluster ## Overview AWS ParallelCluster is an open-source cluster management tool that deploys and manages HPC clusters on AWS. It creates a fully configured Slurm cluster from a YAML configuration file, handling EC2 instances, networking, shared storage, and auto-scaling automatically. The latest release is **ParallelCluster 3.15** (bundling Slurm 25.11). Key differentiator from on-prem: **dynamic scaling**. ParallelCluster can scale compute nodes from zero to thousands based on pending jobs, then terminate idle instances to stop incurring costs. --- ## Architecture ``` ┌─────────────────────────────────────────────────────┐ │ AWS VPC │ │ │ │ ┌──────────────┐ ┌───────────────────────────┐ │ │ │ Head Node │ │ Compute Fleet │ │ │ │ (EC2) │ │ │ │ │ │ slurmctld │◄──►│ Static nodes (always on) │ │ │ │ slurmdbd │ │ Dynamic nodes (scale-to-0)│ │ │ │ MySQL │ │ │ │ │ │ sackd │ │ queue1-st-c5xlarge-[1-N] │ │ │ └──────┬───────┘ │ queue1-dy-c5xlarge-[1-M] │ │ │ │ └───────────────────────────┘ │ │ │ │ │ ┌──────▼───────┐ ┌───────────────────────────┐ │ │ │ Shared │ │ Login Node (optional) │ │ │ │ Storage │ │ User SSH access │ │ │ │ FSx / EFS │ └───────────────────────────┘ │ │ └──────────────┘ │ └─────────────────────────────────────────────────────┘ ``` ### What ParallelCluster Manages | Component | ParallelCluster Handles | You Configure | |-----------|------------------------|---------------| | **Slurm** | Installation, daemons, slurm.conf basics | CustomSlurmSettings, QOS, accounts | | **Compute nodes** | Launch, terminate, health checks | Instance types, MinCount/MaxCount | | **Networking** | VPC, subnets, security groups | Subnet IDs, placement groups | | **Storage** | Mount FSx/EFS/EBS | Storage type, size, throughput | | **Auth** | Munge setup | N/A (managed) | | **Scaling** | Resume/suspend programs | ScaledownIdletime | ### Login Nodes ParallelCluster 3.7.0+ supports managed login node groups. Login nodes use standard configless mode with slurmd to pull configuration from the head node. Users SSH to login nodes to submit and monitor jobs without accessing the head node directly. > **Note:** For multi-cluster scenarios (Slurm 25.05+), sackd can be manually configured on ParallelCluster login nodes to connect to multiple clusters from a single node. This is not a managed ParallelCluster feature -- you would need to create the sackd systemd services and manage the `SLURM_CONF` environment switching yourself. See the [on-premise configless](on-premise.md#multi-cluster-login-nodes-slurm-2505) section for the general sackd multi-cluster pattern. --- ## Cluster Configuration ParallelCluster uses a YAML configuration file. Here is a complete life science cluster example: ```yaml Region: us-east-2 Image: Os: alinux2 HeadNode: InstanceType: c5.2xlarge Networking: SubnetId: subnet-0abc123def456 Ssh: KeyName: my-hpc-key LocalStorage: RootVolume: Size: 100 Iam: AdditionalIamPolicies: - Policy: arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore Scheduling: Scheduler: slurm SlurmSettings: ScaledownIdletime: 10 # Minutes before idle dynamic nodes terminate EnableMemoryBasedScheduling: Enabled: true SlurmQueues: # CPU queue -- genomics, bioinformatics - Name: cpu CapacityType: ONDEMAND ComputeResources: - Name: c5-4xl Instances: - InstanceType: c5.4xlarge # 16 vCPU, 32 GB MinCount: 0 # Scale to zero MaxCount: 100 - Name: c5-18xl Instances: - InstanceType: c5.18xlarge # 72 vCPU, 144 GB MinCount: 0 MaxCount: 20 Networking: SubnetIds: - subnet-0abc123def456 # High-memory queue -- genome assembly, large alignments - Name: highmem CapacityType: ONDEMAND ComputeResources: - Name: r5-12xl Instances: - InstanceType: r5.12xlarge # 48 vCPU, 384 GB MinCount: 0 MaxCount: 10 Networking: SubnetIds: - subnet-0abc123def456 # GPU queue -- cryo-EM, AlphaFold, ML training - Name: gpu CapacityType: ONDEMAND ComputeResources: - Name: g5-12xl Instances: - InstanceType: g5.12xlarge # 4x A10G, 48 vCPU, 192 GB MinCount: 0 MaxCount: 8 - Name: p4d-24xl Instances: - InstanceType: p4d.24xlarge # 8x A100, 96 vCPU, 1.1 TB MinCount: 0 MaxCount: 4 Networking: SubnetIds: - subnet-0abc123def456 PlacementGroup: Enabled: true SharedStorage: - MountDir: /data Name: fsx-lustre StorageType: FsxLustre FsxLustreSettings: StorageCapacity: 7200 # GB (must be 1200 or increments of 2400) DeploymentType: PERSISTENT_2 PerUnitStorageThroughput: 250 # MB/s per TB - MountDir: /home Name: efs-home StorageType: Efs EfsSettings: ThroughputMode: elastic ``` --- ## Static vs. Dynamic Nodes This is the most important ParallelCluster concept for Slurm administrators. ### Static Nodes (always running) - Created at cluster launch, persist until cluster update or deletion - Named `<queue>-st-<resource>-<N>` (e.g., `cpu-st-c5-4xl-1`) - Set with `MinCount` -- these instances are always running and billing - Use for: baseline capacity, always-on workloads ### Dynamic Nodes (scale-to-zero) - Created on demand when jobs are pending, terminated when idle - Named `<queue>-dy-<resource>-<N>` (e.g., `cpu-dy-c5-4xl-1`) - Available count = `MaxCount - MinCount` - Terminated after `ScaledownIdletime` minutes of inactivity (default: 10) - Use for: burst capacity, cost optimization ### Node States in sinfo ```bash $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST cpu* up infinite 20 idle~ cpu-dy-c5-4xl-[1-20] cpu* up infinite 5 idle cpu-st-c5-4xl-[1-5] gpu up infinite 8 idle~ gpu-dy-g5-12xl-[1-8] ``` | State Suffix | Meaning | |-------------|---------| | `idle` | Static node, running, no jobs | | `idle~` | Dynamic node, **not running** (powered down) | | `alloc` | Running a job | | `mix` | Partially allocated | | `idle#` | Dynamic node, **powering up** | | `down` | Node failed health check | The `~` suffix is critical -- it means the EC2 instance does not exist yet and costs nothing. ### Scaling Timeline 1. User submits job: `sbatch --partition=gpu myjob.sh` 2. slurmctld sees pending job, sets dynamic node to `idle#` (powering up) 3. ParallelCluster launches EC2 instance (~2-5 minutes for standard instances) 4. slurmd starts, node transitions to `alloc` 5. Job runs 6. Job completes, node becomes `idle` 7. After `ScaledownIdletime` (default 10 min) with no new jobs, node terminates 8. Node returns to `idle~` (powered down) --- ## Slurm Configuration Customization ParallelCluster generates `slurm.conf` automatically. You can customize it via `CustomSlurmSettings`, but some parameters are on a deny list. ### CustomSlurmSettings (cluster-level) ```yaml Scheduling: SlurmSettings: CustomSlurmSettings: - SchedulerParameters: bf_max_job_test=5000,bf_interval=30 - AccountingStorageEnforce: associations,limits,qos - PriorityType: priority/multifactor - PriorityWeightFairshare: 10000 - PriorityWeightAge: 1000 - PriorityWeightQOS: 5000 ``` ### CustomSlurmSettings (queue-level) ```yaml SlurmQueues: - Name: gpu CustomSlurmSettings: - MaxTime: 3-00:00:00 - DefaultTime: 04:00:00 - PreemptMode: REQUEUE ``` ### Deny-Listed Parameters ParallelCluster blocks certain slurm.conf parameters to prevent conflicts with its scaling logic: | Blocked Parameters | Reason | |-------------------|--------| | SlurmctldHost, SlurmUser | Managed by ParallelCluster | | ResumeProgram, SuspendProgram | Controls auto-scaling | | ResumeTimeout, SuspendTime | Scaling timing | | Prolog, Epilog | Use ParallelCluster's prolog/epilog hooks | | Nodes, PartitionName | Derived from YAML config | | AccountingStorageType/Host | Managed locally on head node | Use `CustomSlurmSettingsIncludeFile` for an external file (bypasses deny list validation, use with caution). --- ## Storage Options ParallelCluster supports multiple shared storage types: Amazon EBS, Amazon EFS, FSx for Lustre, FSx for ONTAP, and FSx for OpenZFS. ### FSx for Lustre Best for: high-throughput parallel I/O (cryo-EM particle data, large genomics datasets) ```yaml SharedStorage: - MountDir: /scratch StorageType: FsxLustre FsxLustreSettings: StorageCapacity: 7200 DeploymentType: PERSISTENT_2 # Durable, multi-AZ PerUnitStorageThroughput: 250 # MB/s per TB ``` - PERSISTENT_2: data survives filesystem deletion, replicated - SCRATCH_2: higher throughput, data lost if filesystem deleted (temp workloads) - Can link to S3 for data import/export - **EFA-enabled FSx Lustre** (ParallelCluster 3.13+): enables higher throughput via Elastic Fabric Adapter for storage traffic ### EFS (Elastic File System) Best for: home directories, software installations, shared config ```yaml SharedStorage: - MountDir: /home StorageType: Efs EfsSettings: ThroughputMode: elastic # Scales automatically ``` - NFS-compatible, fully managed - Lower throughput than FSx Lustre - Good for small-file workloads ### FSx for NetApp ONTAP Best for: multi-protocol access (NFS + SMB), data tiering, snapshots ```yaml SharedStorage: - MountDir: /data StorageType: FsxOntap FsxOntapSettings: VolumeId: fsvol-0abc123def456 ``` - Multi-protocol (NFS, SMB, iSCSI) - Built-in data tiering to S3 - Useful when researchers need both Linux and Windows access ### EBS (Elastic Block Store) Best for: head node local storage only ```yaml HeadNode: LocalStorage: RootVolume: Size: 200 VolumeType: gp3 ``` - Attached to a single instance - Not shared -- only use for head node or as local scratch on compute nodes --- ## Networking ### EFA (Elastic Fabric Adapter) For MPI-heavy workloads (molecular dynamics, quantum chemistry), enable EFA: ```yaml SlurmQueues: - Name: mpi ComputeResources: - Name: hpc6a-48xl Instances: - InstanceType: hpc6a.48xlarge # EFA-enabled HPC instance MinCount: 0 MaxCount: 16 Efa: Enabled: true Networking: SubnetIds: - subnet-0abc123def456 PlacementGroup: Enabled: true # Co-locate instances for low latency ``` ### Placement Groups Enable for latency-sensitive workloads: ```yaml Networking: PlacementGroup: Enabled: true # Cluster placement group ``` Placement groups ensure instances launch in close physical proximity, reducing network latency. Essential for multi-node MPI jobs. --- ## Instance Types for Life Science | Workload | Recommended Instance | vCPU | Memory | GPU | Notes | |----------|---------------------|------|--------|-----|-------| | Genomics (BLAST, BWA) | c5.4xlarge - c5.18xlarge | 16-72 | 32-144 GB | - | CPU-bound | | Genome assembly | r5.12xlarge - r5.24xlarge | 48-96 | 384-768 GB | - | Memory-bound | | Cryo-EM (RELION) | g5.12xlarge | 48 | 192 GB | 4x A10G | GPU-accelerated | | Cryo-EM (CryoSPARC) | p4d.24xlarge | 96 | 1.1 TB | 8x A100 | Multi-GPU | | AlphaFold | g5.xlarge - g5.12xlarge | 4-48 | 16-192 GB | 1-4x A10G | GPU-bound | | ML training | p4d.24xlarge | 96 | 1.1 TB | 8x A100 | Large-scale GPU | | Molecular dynamics | hpc6a.48xlarge | 96 | 384 GB | - | EFA for MPI | --- ## Cost Management ### Scale-to-Zero Pattern The most impactful cost optimization: set `MinCount: 0` for all queues. ```yaml ComputeResources: - Name: c5-4xl MinCount: 0 # No instances running when idle MaxCount: 100 # Can burst to 100 ``` Users experience a 2-5 minute startup delay for the first job, but pay nothing when the cluster is idle. ### Spot Instances For fault-tolerant workloads (job arrays, parameter sweeps), use Spot: ```yaml SlurmQueues: - Name: spot-cpu CapacityType: SPOT AllocationStrategy: capacity-optimized ComputeResources: - Name: spot-compute Instances: - InstanceType: c5.4xlarge - InstanceType: c5a.4xlarge - InstanceType: c5n.4xlarge # Multiple types for availability MinCount: 0 MaxCount: 200 ``` - Spot can save 60-90% vs. On-Demand - Instances can be reclaimed with 2-minute warning - Use with `--requeue` so interrupted jobs restart automatically ### ScaledownIdletime Balance responsiveness vs. cost: ```yaml SlurmSettings: ScaledownIdletime: 10 # Default: terminate after 10 min idle ``` | Value | Trade-off | |-------|-----------| | 2-5 min | Aggressive savings, but frequent launch delays for bursty users | | 10 min (default) | Good balance for most life science workloads | | 30-60 min | Keep warm for interactive users, higher cost | --- ## Cluster Lifecycle ### Create ```bash # Install the CLI pip install aws-parallelcluster # Create cluster from config pcluster create-cluster --cluster-name biocluster \ --cluster-configuration cluster-config.yaml # Monitor creation pcluster describe-cluster --cluster-name biocluster ``` ### Update ```bash # Update config (some changes require compute fleet stop) pcluster update-cluster --cluster-name biocluster \ --cluster-configuration updated-config.yaml # For changes that require fleet stop: pcluster update-compute-fleet --cluster-name biocluster --status STOP_REQUESTED # ... then update, then: pcluster update-compute-fleet --cluster-name biocluster --status START_REQUESTED ``` Changes that do NOT require fleet stop: - Adding new queues or compute resources - Increasing MaxCount - Increasing MinCount (with matching MaxCount increase) - Updating CustomSlurmSettings ### SSH Access ```bash pcluster ssh --cluster-name biocluster -i ~/my-hpc-key.pem ``` ### Delete ```bash pcluster delete-cluster --cluster-name biocluster ``` --- ## Post-Creation Slurm Setup After cluster creation, configure Slurm accounting (not managed by ParallelCluster YAML): ```bash # SSH to head node pcluster ssh --cluster-name biocluster # Create accounts and users sudo sacctmgr add account research Description="Research" sudo sacctmgr add account cryoem parent=research Description="Cryo-EM" sudo sacctmgr add user jsmith Account=research # Create QOS sudo sacctmgr add qos gpu_priority Priority=100 MaxTRESPerUser=gpu=8 sudo sacctmgr add qos burst MaxSubmitJobsPerUser=500 ``` ### Why Accounting Matters on ParallelCluster > **Best Practice:** ParallelCluster deploys slurmdbd on the head node by default, but accounting is only useful once you create accounts, users, and QOS entries (as shown above). Always complete this post-creation setup. Without accounting fully configured, the following capabilities are **unavailable**: > > - **Fairshare scheduling** (priority/multifactor) -- cannot weight user/group priority by historical usage > - **QOS enforcement** -- cannot set resource limits per user, account, or group > - **Job history and reporting** (`sacct`, `sreport`) -- no post-job resource usage data > - **License-aware scheduling** -- cannot integrate FLEXlm license counts (required for Schrodinger and other commercial software) > - **Account-based usage tracking and chargeback** -- no way to attribute compute hours to grants or departments > - **Resource limit enforcement** (MaxJobs, MaxSubmitJobs, GrpTRES) -- cannot prevent any single user from consuming the entire cluster > > On cloud clusters where every compute hour has a direct dollar cost, accounting is arguably more important than on-prem. Without it, you have no mechanism to enforce spend limits per group, no fairshare to balance usage across teams, and no historical data for cost allocation. > **ParallelCluster Note:** The `AccountingStorageType` and `AccountingStorageHost` parameters are on the ParallelCluster deny list because ParallelCluster manages slurmdbd on the head node automatically. You do not need to set these -- just focus on creating accounts, users, and QOS via `sacctmgr`. --- ## Key Differences from On-Prem | Aspect | On-Prem | ParallelCluster | |--------|---------|-----------------| | **Slurm version** | You choose | Bundled with PC version | | **slurm.conf** | Full control | Generated + CustomSlurmSettings | | **HA** | slurmctld primary/backup | Single head node (no native HA) | | **Node scaling** | Fixed | Dynamic (scale-to-zero) | | **Node names** | You define | Auto-generated (`queue-st/dy-resource-N`) | | **GRES/GPU** | Manual gres.conf | Auto-detected from instance type | | **Storage** | Your infrastructure | FSx Lustre, EFS, EBS | | **Networking** | IB, Ethernet | EFA, VPC | | **Prolog/Epilog** | Direct in slurm.conf | ParallelCluster hooks (deny-listed) | | **OS** | Any Linux | Amazon Linux 2, Ubuntu, RHEL | | **Upgrades** | In-place | New cluster (blue-green) | --- ## Troubleshooting ### Nodes stuck in `idle#` (powering up) ```bash # Check cloud node status scontrol show node cpu-dy-c5-4xl-1 # Check ParallelCluster logs sudo tail -f /var/log/parallelcluster/clustermgtd.log sudo tail -f /var/log/parallelcluster/slurm_resume.log ``` Common causes: insufficient capacity in the AZ, service quotas exceeded, subnet out of IPs. ### Nodes going to `down` state ```bash # Check why scontrol show node cpu-dy-c5-4xl-1 | grep Reason # ParallelCluster health check logs sudo tail -f /var/log/parallelcluster/computemgtd.log ``` ### Jobs pending with `(ReqNodeNotAvail)` Normal for dynamic nodes -- the node is launching. Wait 2-5 minutes. If persistent: ```bash # Check if it's a capacity issue squeue -j JOBID -o "%i %j %T %r" # Check EC2 service quotas aws service-quotas get-service-quota \ --service-code ec2 \ --quota-code L-1216C47A # Running On-Demand Standard instances ```

Related Modules

• On-Premise Deployment -- traditional deployment for comparison
• AWS PCS -- AWS managed alternative
• GPU Jobs -- user guide for GPU workloads on the cluster
• Accounts & Fairshare -- post-creation accounting setup
• Cost Allocation -- chargeback with cloud cost attribution

References

• AWS ParallelCluster Documentation
• AWS ParallelCluster Slurm Workload Manager
• Slurm Configuration Customization
• Cluster Capacity Size and Update
• Multiple Instance Type Allocation
• SchedMD: Slurm Power Saving Guide