AWS PCS

<!-- audience: L4 --> <!-- deployment: aws-pcs --> <!-- sge-source: none --> # AWS PCS (Parallel Computing Service) ## Overview AWS Parallel Computing Service (PCS) is a fully managed service that deploys and manages Slurm clusters on AWS. Unlike ParallelCluster (which is an open-source tool you run yourself), PCS is an AWS-native service where AWS manages the Slurm control plane for you. PCS launched in GA in August 2024. It is newer and more opinionated than ParallelCluster, trading configuration flexibility for operational simplicity. ### PCS vs. ParallelCluster | Aspect | ParallelCluster | PCS | |--------|----------------|-----| | **Type** | Open-source CLI tool | AWS managed service | | **Slurm control plane** | Head node EC2 you manage | AWS-managed (no head node in your account) | | **Slurm version** | Bundled with PC version | AWS-managed, updated by AWS | | **HA** | Single head node (your responsibility) | AWS-managed HA | | **Configuration** | YAML config file | AWS console, CLI, or Terraform | | **slurm.conf control** | CustomSlurmSettings (deny-list) | Custom settings (allow-list, 60+ params) | | **Accounting (slurmdbd)** | On head node, you manage | AWS-managed (opt-in, billed; you never see `slurmdbd.conf`) | | **Scaling** | Dynamic nodes via power-save plugin | Dynamic nodes via AWS-managed scaling | | **Storage** | FSx Lustre, EFS, EBS | FSx Lustre, EFS | | **Networking** | Full VPC control | VPC with security groups | | **Login nodes** | Optional, self-managed | Managed login node groups | | **Custom AMI** | Full support | Custom AMI support | | **Cost** | EC2 for head node + compute | No head node cost + compute | | **Best for** | Full Slurm control, complex configs | Simpler ops, less Slurm admin | **Choose ParallelCluster when:** you need full slurm.conf control, complex QOS/priority policies, federation, or specific Slurm versions. **Choose PCS when:** you want minimal Slurm administration, AWS-managed HA, and are comfortable with AWS managing the control plane. --- ## Architecture ``` ┌─────────────────────────────────────────────────────────┐ │ AWS VPC │ │ │ │ ┌──────────────────┐ │ │ │ AWS-Managed │ ┌──────────────────────────┐ │ │ │ Control Plane │ │ Compute Node Groups │ │ │ │ (not in your │◄───►│ │ │ │ │ account) │ │ Static nodes │ │ │ │ │ │ Dynamic nodes │ │ │ │ slurmctld (HA) │ │ (EC2 in your account) │ │ │ │ slurmdbd │ └──────────────────────────┘ │ │ │ MySQL │ │ │ └──────────────────┘ ┌──────────────────────────┐ │ │ │ Login Node Group │ │ │ ┌──────────────────┐ │ (EC2 in your account) │ │ │ │ Shared Storage │ │ User SSH access │ │ │ │ FSx / EFS │ └──────────────────────────┘ │ │ └──────────────────┘ │ └─────────────────────────────────────────────────────────┘ ``` Key difference: the Slurm control plane (slurmctld, slurmdbd, database) runs in AWS-managed infrastructure outside your account. You only see compute nodes and login nodes as EC2 instances. --- ## Core Concepts ### Clusters A PCS cluster is the top-level resource. It defines the Slurm control plane and networking: ```bash aws pcs create-cluster \ --cluster-name biocluster \ --scheduler type=SLURM,version=26.05 \ --networking subnetIds=subnet-0abc123,securityGroupIds=sg-0abc123 \ --size MEDIUM \ --slurm-configuration 'scaleDownIdleTimeInSeconds=600,accounting={mode=STANDARD}' ``` > **Note:** Accounting is opt-in and incurs additional charges (hourly fee + storage fee). Enable it at cluster creation to use `sacct`, `sreport`, QOS enforcement, and chargeback. Retention is set with `defaultPurgeTimeInDays` (default `-1` = keep forever). On Slurm 26.05 PCS sets `CommitDelay=1` by default; if job records go missing after a submission burst, see the PCS troubleshooting topic on `CommitDelay`. > **Best Practice:** Always enable accounting (`accounting={mode=STANDARD}`) when creating a PCS cluster. Despite the additional cost, accounting is the foundation for most of Slurm's advanced scheduling and resource management. Without it, the following capabilities are **unavailable**: > > - **Fairshare scheduling** (priority/multifactor) -- cannot weight user/group priority by historical usage > - **QOS enforcement** -- cannot set resource limits per user, account, or group > - **Job history and reporting** (`sacct`, `sreport`) -- no post-job resource usage data > - **License-aware scheduling** -- cannot integrate FLEXlm license counts (required for Schrodinger and other commercial software) > - **Account-based usage tracking and chargeback** -- no way to attribute compute hours to grants or departments > - **Resource limit enforcement** (MaxJobs, MaxSubmitJobs, GrpTRES) -- cannot prevent any single user from consuming the entire cluster > > On a pay-per-use cloud cluster, running without accounting means you have no mechanism for cost allocation, no fairshare to prevent a single user from monopolizing expensive GPU instances, and no historical data for budgeting. The additional PCS accounting fee is a fraction of the compute cost it helps you control. Cluster sizes determine control plane capacity: | Size | Max Nodes | Use Case | |------|-----------|----------| | SMALL | ~100 | Development, small teams | | MEDIUM | ~1,000 | Typical life science clusters | | LARGE | ~10,000 | Large-scale production | --- ## Compute Node Groups Compute node groups define the EC2 instances that run your jobs. They map to Slurm partitions. ```bash aws pcs create-compute-node-group \ --cluster-name biocluster \ --compute-node-group-name cpu-nodes \ --instance-configs '[ {"instanceType": "c7i.4xlarge"}, {"instanceType": "c7i.8xlarge"} ]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 100 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type AL2023 ``` --- ## Login Node Groups Dedicated login nodes where users SSH in to submit jobs: ```bash aws pcs create-login-node-group \ --cluster-name biocluster \ --login-node-group-name login \ --instance-type c7i.xlarge \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSLoginRole" ``` --- ## Life Science Cluster Example ### Step 1: Create the Cluster ```bash aws pcs create-cluster \ --cluster-name lifesci-cluster \ --scheduler type=SLURM,version=26.05 \ --networking subnetIds=subnet-0abc123,securityGroupIds=sg-0abc123 \ --size MEDIUM \ --slurm-configuration 'scaleDownIdleTimeInSeconds=600,accounting={mode=STANDARD}' ``` --- ## Life Science Cluster Example (continued) ### Step 2: Create Compute Node Groups ```bash # CPU partition -- genomics, bioinformatics aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name cpu \ --instance-configs '[ {"instanceType": "c7i.4xlarge"}, {"instanceType": "c7i.8xlarge"} ]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 50 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type AL2023 # High-memory partition -- genome assembly aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name highmem \ --instance-configs '[{"instanceType": "r7i.12xlarge"}]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 10 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type AL2023 # GPU partition -- cryo-EM, AlphaFold aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name gpu \ --instance-configs '[ {"instanceType": "g6.12xlarge"}, {"instanceType": "p5.48xlarge"} ]' \ --scaling-configuration '{ "minInstanceCount": 0, "maxInstanceCount": 8 }' \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSComputeRole" \ --ami-type AL2023 ``` --- ## Life Science Cluster Example (continued) ### Step 3: Create Login Nodes ```bash aws pcs create-login-node-group \ --cluster-name lifesci-cluster \ --login-node-group-name login \ --instance-type c7i.xlarge \ --subnet-ids '["subnet-0abc123"]' \ --iam-instance-profile-arn "arn:aws:iam::123456789012:instance-profile/PCSLoginRole" ``` ### Step 4: Attach Storage ```bash # FSx for Lustre -- high-performance scratch aws pcs create-cluster \ ... \ --slurm-configuration '{ "slurmCustomSettings": [ {"parameterName": "...", "parameterValue": "..."} ] }' ``` Storage is configured by mounting FSx or EFS filesystems on compute and login nodes via launch templates or custom AMIs. --- ## Scaling Behavior PCS uses the same static/dynamic node model as ParallelCluster: - **Static nodes** (`minInstanceCount > 0`): always running - **Dynamic nodes** (up to `maxInstanceCount`): launched on demand, terminated when idle ```bash # Check cluster state ssh login-node sinfo # PARTITION AVAIL TIMELIMIT NODES STATE NODELIST # cpu* up infinite 50 idle~ cpu-dy-[1-50] # highmem up infinite 10 idle~ highmem-dy-[1-10] # gpu up infinite 8 idle~ gpu-dy-[1-8] ``` The `idle~` state means the instances are powered down and not billing for compute. ### Scaling Timeline Same as ParallelCluster: 1. Job submitted → slurmctld identifies needed nodes 2. PCS launches EC2 instances (~2-5 minutes) 3. slurmd registers with control plane 4. Job dispatched to nodes 5. After idle timeout, PCS terminates instances --- ## User Experience From the user's perspective, PCS clusters feel identical to any Slurm cluster: ```bash # SSH to login node ssh user@login-node-ip # Submit jobs sbatch --partition=cpu --cpus-per-task=16 --mem=32G myjob.sh sbatch --partition=gpu --gres=gpu:4 cryo_em.sh # Monitor squeue -u $USER sinfo sacct -j 12345 --format=JobID,Elapsed,MaxRSS,ExitCode ``` All standard Slurm commands work. The main difference users notice is the startup delay when dynamic nodes need to launch. --- ## What PCS Manages for You | Component | Admin Responsibility | |-----------|---------------------| | **slurmctld** | AWS manages -- HA, upgrades, patches | | **slurmdbd + MySQL** | AWS manages -- backups, scaling | | **Munge/auth** | AWS manages | | **slurm.conf (core)** | AWS manages -- you customize via Slurm settings | | **Compute nodes** | You define instance types, scaling ranges | | **Login nodes** | You define instance type and count | | **Storage** | You attach FSx/EFS via mount configs | | **User identity** | You manage (LDAP, SSSD, or local accounts via custom AMI) | | **Slurm accounting (accounts/QOS)** | You configure via sacctmgr on login node | | **Software/modules** | You install via custom AMI or shared storage | --- ## Slurm Configuration PCS uses an **allow-list** approach to custom Slurm settings -- over 60 parameters are supported at three levels: cluster, queue, and compute node group. This is the opposite of ParallelCluster's deny-list approach. ### Cluster-Level Settings ```bash aws pcs update-cluster \ --cluster-identifier lifesci-cluster \ --slurm-configuration \ 'SlurmCustomSettings=[ {parameterName=PriorityType,parameterValue=priority/multifactor}, {parameterName=PriorityWeightFairshare,parameterValue=10000}, {parameterName=PriorityWeightAge,parameterValue=1000}, {parameterName=Prolog,parameterValue=/shared/scripts/prolog.sh} ]' ``` --- ## Slurm Configuration: Queue and Node Group ### Queue-Level Settings ```bash aws pcs update-queue \ --cluster-identifier lifesci-cluster \ --queue-identifier gpu \ --slurm-configuration \ 'SlurmCustomSettings=[ {parameterName=MaxTime,parameterValue=3-00:00:00}, {parameterName=Default,parameterValue=NO}, {parameterName=PreemptMode,parameterValue=REQUEUE} ]' ``` ### Compute Node Group Settings ```bash aws pcs update-compute-node-group \ --cluster-identifier lifesci-cluster \ --compute-node-group-identifier gpu-a100 \ --slurm-configuration \ 'SlurmCustomSettings=[{parameterName=Features,parameterValue="gpu,a100,nvme"}]' ``` --- ## Slurm Configuration (continued) ### Additional Customization PCS also supports: - **Custom cgroup settings** -- CPU binding, memory limits, device access control - **Custom SlurmDBD settings** -- privacy controls, data retention, workload tracking - **SPANK plugins** -- extend Slurm functionality with custom plugins - **CLI filter plugins** -- customize job submission validation - **Slurm REST API** -- programmatic job management ### Post-Creation Accounting Setup From a login node (requires accounting enabled at cluster creation): ```bash # Create account hierarchy sudo sacctmgr add account research Description="Research groups" sudo sacctmgr add account cryoem parent=research Description="Cryo-EM team" # Add users sudo sacctmgr add user jsmith Account=research # Create QOS sudo sacctmgr add qos gpu_priority Priority=100 MaxTRESPerUser=gpu=8 ``` --- ## Custom AMIs For life science workloads, you almost always need a custom AMI with your software stack: 1. Launch a base PCS-compatible AMI 2. Install your software (modules, Singularity, CUDA, etc.) 3. Create an AMI from the instance 4. Reference it in your compute node group ```bash aws pcs create-compute-node-group \ --cluster-name lifesci-cluster \ --compute-node-group-name gpu \ --custom-ami-id ami-0abc123def456 \ ... ``` Common software to pre-install: - Environment Modules or Lmod - Singularity/Apptainer - NVIDIA drivers + CUDA toolkit (GPU nodes) - Lustre client (`lustre-client` package) - LDAP/SSSD client for identity --- ## Ephemeral Storage and `/scratch` on PCS PCS doesn't auto-mount ephemeral storage the way ParallelCluster does — PCS is thinner and hands you the raw compute node, leaving storage layout up to your launch template or custom AMI. That means on PCS, **`/scratch` only exists if you configure it**, and you make that choice per compute node group. Because PCS compute node groups are EC2 Auto Scaling Groups backed by a launch template, the pattern is: 1. Pick an instance type — with or without local NVMe — based on how much scratch I/O the job needs (same selection table as the ParallelCluster chapter). 2. In the launch template's user-data (or in a custom AMI), format and mount the NVMe at `/scratch` on boot. 3. Optionally add FSx for Lustre to the same compute node group via the `storageConfigurations` block when creating the group, for a shared scratch alongside the per-node ephemeral. ### Example user-data to mount NVMe at /scratch ```bash #!/bin/bash # Instance Store NVMe devices appear as /dev/nvme1n1, /dev/nvme2n1, ... # /dev/nvme0n1 is always the root EBS volume — skip it. set -euo pipefail EPHEMERAL_DEVICES=$(lsblk -dno NAME,TYPE,MOUNTPOINT | \ awk '$1 ~ /^nvme[1-9]/ && $2 == "disk" && $3 == "" { print "/dev/"$1 }') if [ -n "$EPHEMERAL_DEVICES" ]; then # One device: mkfs+mount directly. Multiple devices: RAID0 across them. DEVICE_COUNT=$(echo "$EPHEMERAL_DEVICES" | wc -l) if [ "$DEVICE_COUNT" -gt 1 ]; then mdadm --create /dev/md0 --level=0 --raid-devices=$DEVICE_COUNT $EPHEMERAL_DEVICES TARGET=/dev/md0 else TARGET=$EPHEMERAL_DEVICES fi mkfs.ext4 -F $TARGET mkdir -p /scratch mount $TARGET /scratch chmod 1777 /scratch fi ``` This reproduces what ParallelCluster does automatically. You only need it on instance families with instance storage (g4dn, g6, c6id, m6id, i4i, etc.). On `t3`/`c6g`/etc. without instance storage, there is no ephemeral disk and `/scratch` simply doesn't get created — your jobs either don't need scratch or use a shared filesystem. ### Two scratch tiers: ephemeral + shared The common production pattern on PCS mirrors the on-prem "hot cache" plus "warm scratch" layout: | Mount | Backend | Purpose | Lifetime | |-------|---------|---------|----------| | `/scratch` | Local NVMe (instance store) | Per-job staging, small-file I/O | Node lifetime | | `/fsx` or `/shared-scratch` | FSx for Lustre | Cross-node coordination, shared intermediates | Filesystem lifetime | | `/home` | EFS | User home directories, small shared files | Persistent | | `/software` | EFS | Application installations, modulefiles | Persistent | Job scripts stage inputs: shared filesystem → local `/scratch` → compute against local → copy results back to shared. Any job that needs cross-node data visibility uses `/fsx` directly and skips the stage-in step. ### Key differences from ParallelCluster | Concern | ParallelCluster | PCS | |---------|-----------------|-----| | Ephemeral auto-mount | Yes, at `/scratch` by default | No — you wire it up in user-data or custom AMI | | RAID0 across multiple NVMe | Automatic | Manual (mdadm in user-data) | | Changing mount point | `EphemeralVolume.MountDir` in cluster config | Edit the launch template | | Scope | Cluster-wide default with per-queue overrides | Per compute node group | ### Portable code: `$SLURM_TMPDIR` PCS's managed Slurm sets `TmpFS` appropriately for each node, so `$SLURM_TMPDIR` points at per-job scratch regardless of whether your node has instance storage or is falling back to root-volume scratch. Prefer `$SLURM_TMPDIR` over hard-coded `/scratch/$USER/$SLURM_JOB_ID` paths — the variable is portable across pcluster, PCS, and on-prem. --- ## Storage Patterns ### FSx for Lustre (high-throughput data) Mount via custom AMI `/etc/fstab` or post-boot script: ```bash # In custom AMI or bootstrap script mount -t lustre fs-0abc123.fsx.us-east-2.amazonaws.com@tcp:/fsx /scratch ``` ### EFS (home directories, software) ```bash mount -t nfs4 fs-0abc123.efs.us-east-2.amazonaws.com:/ /home ``` ### Storage Layout Recommendation ``` /home/ EFS -- user home directories /software/ EFS -- shared application installations, module files /scratch/ FSx Lustre -- high-throughput job I/O ``` --- ## Networking ### Security Groups Configure security groups to allow: - Slurm daemon communication between compute nodes (TCP 6817-6819) - PCS control plane → compute nodes - SSH from login nodes to compute nodes - Storage access (Lustre TCP 988, NFS TCP 2049) ### EFA For MPI workloads, enable EFA on compute node groups by selecting EFA-capable instance types (hpc6a, hpc7a, p5, etc.) and enabling EFA in the launch template. --- ## Supported Slurm Versions AWS manages Slurm versions and applies patch updates automatically. Supported as of October 2026 (check the [PCS Slurm versions page](https://docs.aws.amazon.com/pcs/latest/userguide/slurm-versions.html) before creating a cluster -- the list moves every six months): | Slurm Version | PCS Release Date | EOL Date | |--------------|-----------------|----------| | **26.05** | September 2026 | 2027-11-30 | | **25.11** | April 2026 | 2027-05-31 | | **25.05** | October 2025 | 2026-11-30 | | 24.11 | May 2025 | unsupported since 2026-05-31 | SchedMD supports a release for 18 months, so three versions overlap at any time. When a version reaches EOL no new clusters can use it; existing clusters keep running but should be moved. Since 2026-06 PCS supports **in-place scheduler updates**: a cluster on 24.11, 25.05 or 25.11 can be updated directly to 26.05 (clusters that use Slurm CLI-filter plugins cannot do rolling updates from versions before 25.11). From 26.05 the PCS sample AMI is version-agnostic -- one AL2023 image covers every non-EOL Slurm version. > **Note:** Slurm 25.05 added multi-cluster sackd configuration (login nodes can serve multiple clusters without restart). 25.11 turned on expedited requeue by default and added a `requeue_delay` (5 s) for instance launch failures; a partition `AllowQOS`/`DenyQOS` that names a non-existent QOS makes `slurmctld` exit fatally on 25.11 -- a documented known issue. 26.05 on PCS sets `AutoDetect=full` on GPU node groups by default, adds the `Exclusive` queue setting (`ExclusiveUser` deprecated), takes the REST JWT username from the `id.name` claim, and requires removing `SchedulerParameters=enable_job_state_cache` before upgrading. --- ## Slurm 26.05 on PCS PCS has run 26.05 since 2026-09-10. No ParallelCluster release ships it, so this section is prose, not a validated exercise. **26.05+:** marks behaviour that needs 26.05; the rest needs 25.11. - **26.05+: GPU autodetect is on by default.** On GPU compute node groups PCS sets `AutoDetect=full` when the node's socket layout is known (you set `Sockets` in the node group's custom Slurm settings, or PCS detected the CPU topology). Otherwise it emits its GPU record without `AutoDetect`. Contrast [ParallelCluster](aws-parallelcluster.md#how-parallelcluster-manages-gpu-gres), where `AutoDetect` is rejected on cloud nodes. - **Custom GRES is a pair of settings.** `gresCustomSettings` holds `gres.conf` records for one node group (PCS adds the `NodeName=` prefix); the `Gres` custom Slurm setting declares what the scheduler may allocate. A `Gres` entry with no `gres.conf` record behind it makes the node report fewer resources and drain; so does a record whose `File`/`Cores`/`Links` do not match the booted hardware. PCS validates syntax and consistency, not hardware. Check with `scontrol show node <n> | grep -E 'State|Reason'` (`INVALID_REG`). - **26.05+: `Exclusive` queue setting** (`NO`, `NODE`, `USER`, `TOPO`) replaces `ExclusiveUser` and `OverSubscribe=EXCLUSIVE`. Both old forms still work on PCS; migrate. - **26.05+: REST API identity.** `slurmrestd` reads the username from `name` inside the JWT `id` claim. The top-level `sun`, `username` and custom `userclaimfield` claims are deprecated; existing tokens still authenticate. Put the name in `id.name` now: PCS accepts it on every supported version. - **Before you update to 26.05:** remove `SchedulerParameters=enable_job_state_cache` from the cluster's custom settings (Slurm removed it). 26.05 also sets `CommitDelay=1` in `slurmdbd.conf` by default. ```bash aws pcs update-compute-node-group \ --cluster-identifier lifesci-cluster --compute-node-group-identifier gpu \ --slurm-configuration 'gresCustomSettings=[{Name=gpu,Type=l4,File=/dev/nvidia[0-3]}],slurmCustomSettings=[{parameterName=Gres,parameterValue=gpu:l4:4}]' ``` --- ## Operating a 26.05 Cluster on PCS - **Login nodes** run `sackd` against the cluster controller. One login node can serve several PCS clusters, including clusters in other Regions: see [Multi-Cluster Login Nodes](#multi-cluster-login-nodes). - **Node lifecycle actions** (PCS agent 1.5.0 and later) run scripts at two stages: `nodeBootstrapped` (before `slurmd` starts) and `nodeReady` (after the node registers). They replace launch-template user data for mounting storage and forwarding logs. Changing them drains and replaces nodes, and a script must not reboot the instance. Package installs belong in the custom AMI, for the [same reason as on ParallelCluster](aws-parallelcluster.md#measuring-and-shrinking-cold-start). - **Scheduler audit logs** (25.11+) use their own `PCS_SCHEDULER_AUDIT_LOGS` log type. They are opt-in, take up to three destinations, and can be up to 90% of scheduler log volume: turn them on deliberately. - **Managed Prometheus collector** (25.11+): enable Slurm metrics with the `MetricsType` and `CommunicationParameters` custom settings, then create a scraper with `aws amp create-scraper`. It delivers to an Amazon Managed Service for Prometheus workspace or a CloudWatch dataset and reaches the controller on port 6817. --- ## Multi-Cluster Login Nodes PCS supports configuring a single login node to access multiple PCS clusters using the multi-cluster sackd feature introduced in Slurm 25.05. This is useful when an organization runs multiple PCS clusters (for example, separate clusters for different departments, projects, or security boundaries) and wants users to access all of them from one login node. ### How It Works Each PCS cluster runs its own Slurm control plane. On the shared login node, a separate sackd daemon runs for each cluster, connecting to that cluster's slurmctld. Users activate a specific cluster environment before running Slurm commands. ### Automated Setup PCS provides a setup script, `pcs-multi-cluster-login-configure.sh`, that automates the configuration: - Retrieves authentication keys for each cluster from AWS Secrets Manager - Creates cluster-specific systemd service files for sackd (each with its own `RuntimeDirectory`) - Generates per-cluster activate and deactivate shell scripts After running the script, users switch between clusters with: ```bash # Activate the genomics cluster source /etc/pcs/activate-genomics.sh sinfo # shows genomics cluster partitions squeue # shows genomics cluster queue # Switch to the cryoem cluster source /etc/pcs/deactivate-genomics.sh source /etc/pcs/activate-cryoem.sh sinfo # shows cryoem cluster partitions ``` The activate script sets `SLURM_CONF` to point at the appropriate cluster's cached configuration. The deactivate script unsets it. ### Requirements - **Slurm 25.05+** -- the multi-cluster sackd feature is not available in earlier versions - **AWS Secrets Manager access** -- the login node's IAM role must have permission to read the auth keys for each cluster - **Network connectivity** -- the login node must be able to reach the slurmctld endpoint for each PCS cluster (TCP 6817) - **Shared storage** -- if users need access to job data across clusters, the login node should mount the relevant FSx or EFS filesystems for each cluster ### Considerations This is a PCS-specific convenience on top of the general Slurm 25.05 multi-cluster sackd pattern (see the [on-premise configless](on-premise.md#multi-cluster-login-nodes-slurm-2505) section for the underlying mechanism). The `pcs-multi-cluster-login-configure.sh` script handles the details that would otherwise require manual systemd service file creation and secret management. Users can only interact with one cluster at a time per shell session. Running `squeue` without activating a cluster first will fail or return results from whatever cluster was last activated. --- ## Limitations Current PCS limitations compared to ParallelCluster and on-prem: | Limitation | Impact | Workaround | |-----------|--------|------------| | **Allow-list for settings** | Only approved params can be set | AWS adds more based on feedback | | **No Slurm federation** | Cannot federate with other clusters | Use ParallelCluster instead | | **Slurm version tied to PCS** | Cannot choose arbitrary version | 3 versions available at any time; in-place update to the newest | | **No direct slurmctld access** | Cannot SSH to controller | Use sacctmgr/scontrol from login node | | **Accounting costs extra** | Hourly + storage fees | Only enable if you need it | | **Regional availability** | Not in all AWS regions | Check PCS region availability | --- ## Cost Comparison | Component | ParallelCluster | PCS | |-----------|----------------|-----| | **Control plane** | Head node EC2 (24/7) | No charge (AWS-managed) | | **Compute** | EC2 (same) | EC2 (same) | | **Storage** | FSx/EFS (same) | FSx/EFS (same) | | **Networking** | VPC/data transfer (same) | VPC/data transfer (same) | PCS eliminates the head node EC2 cost, which can be $100-500/month depending on instance type (for scale: a c7i.2xlarge is $0.357/h On-Demand in us-east-2, about $260/month, per the AWS price list of 2026-10-15). For large clusters with many dynamic compute nodes, this savings is marginal. For small clusters, it can be meaningful. --- ## When to Use PCS for Life Science **Good fit:** - Teams that want Slurm without managing the control plane - Straightforward queue structures (CPU, highmem, GPU partitions) - Organizations with AWS expertise but limited Slurm admin experience - Clusters where HA is important but setting up slurmctld failover is too complex **Less ideal:** - Complex Slurm configurations (advanced preemption, federation, custom plugins) - Teams that need pinned Slurm versions for reproducibility - Workflows that require direct slurmctld access or custom prolog/epilog

Related Modules

• On-Premise Deployment -- traditional deployment for comparison
• AWS ParallelCluster -- open-source alternative with more control
• Accounts & Fairshare -- setting up accounting after cluster creation
• Cost Allocation -- chargeback and showback models

References

• AWS PCS Documentation
• AWS PCS API Reference
• AWS PCS Pricing
• AWS ParallelCluster Documentation (for comparison)
• SchedMD: Slurm Power Saving Guide
• PCS: Slurm Versions
• PCS: Release Notes for Slurm Versions (26.05 changes, deprecations, enable_job_state_cache)
• PCS: Custom GRES Settings (gresCustomSettings, Gres pairing, AutoDetect=full)
• PCS: Custom Slurm Settings for Queues (Exclusive)
• PCS: Authenticating with the Slurm REST API (id.name)
• PCS: Node Lifecycle Actions
• PCS: Scheduler Audit Logs
• PCS: Slurm Metrics with a Managed Prometheus Collector
• PCS: Troubleshooting INVALID_REG Nodes