AWS PCS

<!-- audience: L4 --> <!-- deployment: aws-pcs --> <!-- sge-source: none --> # AWS PCS (Parallel Computing Service) ## Overview AWS Parallel Computing Service (PCS) is a fully managed service that deploys and manages Slurm clusters on AWS. Unlike ParallelCluster (which is an open-source tool you run yourself), PCS is an AWS-native service where AWS manages the Slurm control plane for you. PCS launched in GA in August 2024. It is newer and more opinionated than ParallelCluster, trading configuration flexibility for operational simplicity. ### PCS vs. ParallelCluster | Aspect | ParallelCluster | PCS | |--------|----------------|-----| | **Type** | Open-source CLI tool | AWS managed service | | **Slurm control plane** | Head node EC2 you manage | AWS-managed (no head node in your account) | | **Slurm version** | Bundled with PC version | AWS-managed, updated by AWS | | **HA** | Single head node (your responsibility) | AWS-managed HA | | **Configuration** | YAML config file | AWS console, CLI, or Terraform | | **slurm.conf control** | CustomSlurmSettings (deny-list) | Custom settings (allow-list, 60+ params) | | **Accounting (slurmdbd)** | On head node, you manage | AWS-managed (opt-in, Slurm 24.11+) | | **Scaling** | Dynamic nodes via power-save plugin | Dynamic nodes via AWS-managed scaling | | **Storage** | FSx Lustre, EFS, EBS | FSx Lustre, EFS | | **Networking** | Full VPC control | VPC with security groups | | **Login nodes** | Optional, self-managed | Managed login node groups | | **Custom AMI** | Full support | Custom AMI support | | **Cost** | EC2 for head node + compute | No head node cost + compute | | **Best for** | Full Slurm control, complex configs | Simpler ops, less Slurm admin | **Choose ParallelCluster when:** you need full slurm.conf control, complex QOS/priority policies, federation, or specific Slurm versions. **Choose PCS when:** you want minimal Slurm administration, AWS-managed HA, and are comfortable with AWS managing the control plane. --- ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ AWS VPC │ │ │ │ ┌──────────────────┐ │ │ │ AWS-Managed │ ┌──────────────────────────┐ │ │ │ Control Plane │ │ Compute Node Groups │ │ │ │ (not in your │◄───►│ │ │ │ │ account) │ │ Static nodes │ │ │ │ │ │ Dynamic nodes │ │ │ │ slurmctld (HA) │ │ (EC2 in your account) │ │ │ │ slurmdbd │ └──────────────────────────┘ │ │ │ MySQL │ │ │ └──────────────────┘ ┌──────────────────────────┐ │ │ │ Login Node Group │ │ │ ┌──────────────────┐ │ (EC2 in your account) │ │ │ │ Shared Storage │ │ User SSH access │ │ │ │ FSx / EFS │ └──────────────────────────┘ │ │ └──────────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` Key difference: the Slurm control plane (slurmctld, slurmdbd, database) runs in AWS-managed infrastructure outside your account. You only see compute nodes and login nodes as EC2 instances. --- ## Core Concepts ### Clusters A PCS cluster is the top-level resource. It defines the Slurm control plane and networking: ```bash aws pcs create-cluster \ --cluster-name biocluster \ --scheduler type=SLURM,version=24.11 \ --networking subnetIds=subnet-0abc123,securityGroupIds=sg-0abc123 \ --size MEDIUM \ --slurm-configuration 'scaleDownIdleTimeInSeconds=600,accounting={mode=STANDARD}' ``` > **Note:** Accounting is opt-in and incurs additional charges (hourly fee + storage fee). Enable it at cluster creation to use `sacct`, `sreport`, QOS enforcement, and chargeback. Requires Slurm 24.11+. > **Best Practice:** Always enable accounting (`accounting={mode=STANDARD}`) when creating a PCS cluster. Despite the additional cost, accounting is the foundation for most of Slurm's advanced scheduling and resource management. Without it, the following capabilities are **unavailable**: > > - **Fairshare scheduling** (priority/multifactor) -- cannot weight user/group priority by historical usage > - **QOS enforcement** -- cannot set resource limits per user, account, or group > - **Job history and reporting** (`sacct`, `sreport`) -- no post-job resource usage data > - **License-aware scheduling** -- cannot integrate FLEXlm license counts (required for Schrodinger and other commercial software) > - **Account-based usage tracking and chargeback** -- no way to attribute compute hours to grants or departments > - **Resource limit enforcement** (MaxJobs, MaxSubmitJobs, GrpTRES) -- cannot prevent any single user from consuming the entire cluster > > On a pay-per-use cloud cluster, running without accounting means you have no mechanism for cost allocation, no fairshare to prevent a single user from monopolizing expensive GPU instances, and no historical data for budgeting. The additional PCS accounting fee is a fraction of the compute cost it helps you control. Cluster sizes determine control plane capacity: | Size | Max Nodes | Use Case | |------|-----------|----------| | SMALL | ~100 | Development, small teams | | MEDIUM | ~1,000 | Typical life science clusters | | LARGE | ~10,000 | Large-scale production | ### Compute Node Groups Compute node groups define the EC2 instances that run your jobs. They map to Slurm partitions. ```bash aws pcs create-compute-node-group \ --cluster-name biocluster \ --compute-node-group-name cpu-nodes \ --instance-configs '[ {"instanceType": "c5.4xlarge"}, {"instanceType": "c5.9xlarge"} ]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 100 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type ALINUX2 ``` ### Login Node Groups Dedicated login nodes where users SSH in to submit jobs: ```bash aws pcs create-login-node-group \ --cluster-name biocluster \ --login-node-group-name login \ --instance-type c5.xlarge \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSLoginRole" ``` --- ## Life Science Cluster Example ### Step 1: Create the Cluster ```bash aws pcs create-cluster \ --cluster-name lifesci-cluster \ --scheduler type=SLURM,version=24.11 \ --networking subnetIds=subnet-0abc123,securityGroupIds=sg-0abc123 \ --size MEDIUM \ --slurm-configuration 'scaleDownIdleTimeInSeconds=600,accounting={mode=STANDARD}' ``` ### Step 2: Create Compute Node Groups ```bash # CPU partition -- genomics, bioinformatics aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name cpu \ --instance-configs '[ {"instanceType": "c5.4xlarge"}, {"instanceType": "c5.9xlarge"} ]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 50 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type ALINUX2 # High-memory partition -- genome assembly aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name highmem \ --instance-configs '[{"instanceType": "r5.12xlarge"}]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 10 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type ALINUX2 # GPU partition -- cryo-EM, AlphaFold aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name gpu \ --instance-configs '[ {"instanceType": "g5.12xlarge"}, {"instanceType": "p4d.24xlarge"} ]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 8 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type ALINUX2 ``` ### Step 3: Create Login Nodes ```bash aws pcs create-login-node-group \ --cluster-name lifesci-cluster \ --login-node-group-name login \ --instance-type c5.xlarge \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSLoginRole" ``` ### Step 4: Attach Storage ```bash # FSx for Lustre -- high-performance scratch aws pcs create-cluster \ ... \ --slurm-configuration '{ "slurmCustomSettings": [ {"parameterName": "...", "parameterValue": "..."} ] }' ``` Storage is configured by mounting FSx or EFS filesystems on compute and login nodes via launch templates or custom AMIs. --- ## Scaling Behavior PCS uses the same static/dynamic node model as ParallelCluster: - **Static nodes** (`minInstanceCount > 0`): always running - **Dynamic nodes** (up to `maxInstanceCount`): launched on demand, terminated when idle ```bash # Check cluster state ssh login-node sinfo # PARTITION AVAIL TIMELIMIT NODES STATE NODELIST # cpu* up infinite 50 idle~ cpu-dy-[1-50] # highmem up infinite 10 idle~ highmem-dy-[1-10] # gpu up infinite 8 idle~ gpu-dy-[1-8] ``` The `idle~` state means the instances are powered down and not billing for compute. ### Scaling Timeline Same as ParallelCluster: 1. Job submitted → slurmctld identifies needed nodes 2. PCS launches EC2 instances (~2-5 minutes) 3. slurmd registers with control plane 4. Job dispatched to nodes 5. After idle timeout, PCS terminates instances --- ## User Experience From the user's perspective, PCS clusters feel identical to any Slurm cluster: ```bash # SSH to login node ssh user@login-node-ip # Submit jobs sbatch --partition=cpu --cpus-per-task=16 --mem=32G myjob.sh sbatch --partition=gpu --gres=gpu:4 cryo_em.sh # Monitor squeue -u $USER sinfo sacct -j 12345 --format=JobID,Elapsed,MaxRSS,ExitCode ``` All standard Slurm commands work. The main difference users notice is the startup delay when dynamic nodes need to launch. --- ## What PCS Manages for You | Component | Admin Responsibility | |-----------|---------------------| | **slurmctld** | AWS manages -- HA, upgrades, patches | | **slurmdbd + MySQL** | AWS manages -- backups, scaling | | **Munge/auth** | AWS manages | | **slurm.conf (core)** | AWS manages -- you customize via Slurm settings | | **Compute nodes** | You define instance types, scaling ranges | | **Login nodes** | You define instance type and count | | **Storage** | You attach FSx/EFS via mount configs | | **User identity** | You manage (LDAP, SSSD, or local accounts via custom AMI) | | **Slurm accounting (accounts/QOS)** | You configure via sacctmgr on login node | | **Software/modules** | You install via custom AMI or shared storage | --- ## Slurm Configuration PCS uses an **allow-list** approach to custom Slurm settings -- over 60 parameters are supported at three levels: cluster, queue, and compute node group. This is the opposite of ParallelCluster's deny-list approach. ### Cluster-Level Settings ```bash aws pcs update-cluster \ --cluster-identifier lifesci-cluster \ --slurm-configuration \ 'SlurmCustomSettings=[ {parameterName=PriorityType,parameterValue=priority/multifactor}, {parameterName=PriorityWeightFairshare,parameterValue=10000}, {parameterName=PriorityWeightAge,parameterValue=1000}, {parameterName=Prolog,parameterValue=/shared/scripts/prolog.sh} ]' ``` ### Queue-Level Settings ```bash aws pcs update-queue \ --cluster-identifier lifesci-cluster \ --queue-identifier gpu \ --slurm-configuration \ 'SlurmCustomSettings=[ {parameterName=MaxTime,parameterValue=3-00:00:00}, {parameterName=Default,parameterValue=NO}, {parameterName=PreemptMode,parameterValue=REQUEUE} ]' ``` ### Compute Node Group Settings ```bash aws pcs update-compute-node-group \ --cluster-identifier lifesci-cluster \ --compute-node-group-identifier gpu-a100 \ --slurm-configuration \ 'SlurmCustomSettings=[{parameterName=Features,parameterValue="gpu,a100,nvme"}]' ``` ### Additional Customization PCS also supports: - **Custom cgroup settings** -- CPU binding, memory limits, device access control - **Custom SlurmDBD settings** -- privacy controls, data retention, workload tracking - **SPANK plugins** -- extend Slurm functionality with custom plugins - **CLI filter plugins** -- customize job submission validation - **Slurm REST API** -- programmatic job management ### Post-Creation Accounting Setup From a login node (requires accounting enabled at cluster creation): ```bash # Create account hierarchy sudo sacctmgr add account research Description="Research groups" sudo sacctmgr add account cryoem parent=research Description="Cryo-EM team" # Add users sudo sacctmgr add user jsmith Account=research # Create QOS sudo sacctmgr add qos gpu_priority Priority=100 MaxTRESPerUser=gpu=8 ``` --- ## Custom AMIs For life science workloads, you almost always need a custom AMI with your software stack: 1. Launch a base PCS-compatible AMI 2. Install your software (modules, Singularity, CUDA, etc.) 3. Create an AMI from the instance 4. Reference it in your compute node group ```bash aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name gpu \ --custom-ami-id ami-0abc123def456 \ ... ``` Common software to pre-install: - Environment Modules or Lmod - Singularity/Apptainer - NVIDIA drivers + CUDA toolkit (GPU nodes) - Lustre client (`lustre-client` package) - LDAP/SSSD client for identity --- ## Storage Patterns ### FSx for Lustre (high-throughput data) Mount via custom AMI `/etc/fstab` or post-boot script: ```bash # In custom AMI or bootstrap script mount -t lustre fs-0abc123.fsx.us-east-2.amazonaws.com@tcp:/fsx /scratch ``` ### EFS (home directories, software) ```bash mount -t nfs4 fs-0abc123.efs.us-east-2.amazonaws.com:/ /home ``` ### Storage Layout Recommendation ``` /home/ EFS -- user home directories /software/ EFS -- shared application installations, module files /scratch/ FSx Lustre -- high-throughput job I/O ``` --- ## Networking ### Security Groups Configure security groups to allow: - Slurm daemon communication between compute nodes (TCP 6817-6819) - PCS control plane → compute nodes - SSH from login nodes to compute nodes - Storage access (Lustre TCP 988, NFS TCP 2049) ### EFA For MPI workloads, enable EFA on compute node groups by selecting EFA-capable instance types (hpc6a, hpc7a, p4d, p5, etc.) and enabling EFA in the launch template. --- ## Supported Slurm Versions AWS manages Slurm versions and applies patch updates automatically. Currently supported: | Slurm Version | PCS Release Date | EOL Date | |--------------|-----------------|----------| | **25.05** | October 2025 | May 2027 | | **24.11** | May 2025 | May 2026 | SchedMD supports up to 3 major versions at a time. When a version reaches EOL, no new clusters can use it, but existing clusters continue for up to 12 months. Use the latest version for new clusters. > **Note:** Slurm 25.05 adds multi-cluster sackd configuration (login nodes can serve multiple clusters without restart) and improved requeue behavior for instance launch failures. --- ## Multi-Cluster Login Nodes PCS supports configuring a single login node to access multiple PCS clusters using the multi-cluster sackd feature introduced in Slurm 25.05. This is useful when an organization runs multiple PCS clusters (for example, separate clusters for different departments, projects, or security boundaries) and wants users to access all of them from one login node. ### How It Works Each PCS cluster runs its own Slurm control plane. On the shared login node, a separate sackd daemon runs for each cluster, connecting to that cluster's slurmctld. Users activate a specific cluster environment before running Slurm commands. ### Automated Setup PCS provides a setup script, `pcs-multi-cluster-login-configure.sh`, that automates the configuration: - Retrieves authentication keys for each cluster from AWS Secrets Manager - Creates cluster-specific systemd service files for sackd (each with its own `RuntimeDirectory`) - Generates per-cluster activate and deactivate shell scripts After running the script, users switch between clusters with: ```bash # Activate the genomics cluster source /etc/pcs/activate-genomics.sh sinfo # shows genomics cluster partitions squeue # shows genomics cluster queue # Switch to the cryoem cluster source /etc/pcs/deactivate-genomics.sh source /etc/pcs/activate-cryoem.sh sinfo # shows cryoem cluster partitions ``` The activate script sets `SLURM_CONF` to point at the appropriate cluster's cached configuration. The deactivate script unsets it. ### Requirements - **Slurm 25.05+** -- the multi-cluster sackd feature is not available in earlier versions - **AWS Secrets Manager access** -- the login node's IAM role must have permission to read the auth keys for each cluster - **Network connectivity** -- the login node must be able to reach the slurmctld endpoint for each PCS cluster (TCP 6817) - **Shared storage** -- if users need access to job data across clusters, the login node should mount the relevant FSx or EFS filesystems for each cluster ### Considerations This is a PCS-specific convenience on top of the general Slurm 25.05 multi-cluster sackd pattern (see the [on-premise configless](on-premise.md#multi-cluster-login-nodes-slurm-2505) section for the underlying mechanism). The `pcs-multi-cluster-login-configure.sh` script handles the details that would otherwise require manual systemd service file creation and secret management. Users can only interact with one cluster at a time per shell session. Running `squeue` without activating a cluster first will fail or return results from whatever cluster was last activated. --- ## Limitations Current PCS limitations compared to ParallelCluster and on-prem: | Limitation | Impact | Workaround | |-----------|--------|------------| | **Allow-list for settings** | Only approved params can be set | AWS adds more based on feedback | | **No Slurm federation** | Cannot federate with other clusters | Use ParallelCluster instead | | **Slurm version tied to PCS** | Cannot choose arbitrary version | 2 major versions available | | **No direct slurmctld access** | Cannot SSH to controller | Use sacctmgr/scontrol from login node | | **Accounting costs extra** | Hourly + storage fees | Only enable if you need it | | **Regional availability** | Not in all AWS regions | Check PCS region availability | --- ## Cost Comparison | Component | ParallelCluster | PCS | |-----------|----------------|-----| | **Control plane** | Head node EC2 (24/7) | No charge (AWS-managed) | | **Compute** | EC2 (same) | EC2 (same) | | **Storage** | FSx/EFS (same) | FSx/EFS (same) | | **Networking** | VPC/data transfer (same) | VPC/data transfer (same) | PCS eliminates the head node EC2 cost, which can be $100-500/month depending on instance type. For large clusters with many dynamic compute nodes, this savings is marginal. For small clusters, it can be meaningful. --- ## When to Use PCS for Life Science **Good fit:** - Teams that want Slurm without managing the control plane - Straightforward queue structures (CPU, highmem, GPU partitions) - Organizations with AWS expertise but limited Slurm admin experience - Clusters where HA is important but setting up slurmctld failover is too complex **Less ideal:** - Complex Slurm configurations (advanced preemption, federation, custom plugins) - Teams that need pinned Slurm versions for reproducibility - Workflows that require direct slurmctld access or custom prolog/epilog

Related Modules

• On-Premise Deployment -- traditional deployment for comparison
• AWS ParallelCluster -- open-source alternative with more control
• Accounts & Fairshare -- setting up accounting after cluster creation
• Cost Allocation -- chargeback and showback models

References

• AWS PCS Documentation
• AWS PCS API Reference
• AWS PCS Pricing
• AWS ParallelCluster Documentation (for comparison)
• SchedMD: Slurm Power Saving Guide