AWS Troubleshooting

# AWS Slurm Troubleshooting

---

## How AWS Slurm Differs

Running Slurm on AWS introduces failure modes that don't exist on-premise. Nodes are ephemeral, capacity is shared, and network connectivity is dynamic.

This module covers AWS-specific issues. For general Slurm troubleshooting (pending jobs, node states, log files), see the [Troubleshooting](../admin/troubleshooting.md) module.

### Key Differences from On-Premise

- Nodes can **fail to launch** (capacity, quotas, AMI issues)
- Nodes can **disappear mid-job** (Spot interruptions, health checks)
- Nodes must **bootstrap and register** before running jobs (cloud-init, network, DNS)
- **Scaling adds latency** between job submission and execution

---

## Insufficient Capacity Errors

The most common AWS-specific failure. EC2 cannot provide the requested instance type in the requested Availability Zone.

### What You See

```
$ sinfo
PARTITION AVAIL TIMELIMIT NODES STATE  NODELIST
queue1*   up    infinite  1     down#  queue1-dy-c-1-1
queue1*   up    infinite  14    down~  queue1-dy-c-1-[2-15]
```

Nodes in `down#` or `down~` with reason:

```
(Code:InsufficientInstanceCapacity)Failure when resuming nodes.
```

---

## Insufficient Capacity: Error Codes

| Error Code | Meaning |
|-----------|---------|
| `InsufficientInstanceCapacity` | No instances available in this AZ |
| `InsufficientHostCapacity` | Dedicated Host capacity exhausted |
| `MaxSpotInstanceCountExceeded` | Spot instance limit reached |
| `SpotMaxPriceTooLow` | Spot bid below minimum fulfillment price |
| `Unsupported` | Instance type not available in this Region/AZ |
| `VcpuLimitExceeded` | Account vCPU service quota hit |

---

## Insufficient Capacity: Resolution

**Immediate steps:**

```bash
# Check which nodes are affected and why (head node)
sinfo -R
scontrol show nodes queue1-dy-c-1-1 | grep Reason

# Check the resume log for details (head node)
sudo tail -50 /var/log/parallelcluster/slurm_resume.log
```

**Fixes:**

- **Add multiple instance types** to your compute resource — ParallelCluster will fail over to alternatives
- **Use multiple AZs** via subnet configuration
- **Request a service quota increase** for vCPUs (Service Quotas console)
- **Use Capacity Reservations** for critical instance types

> **ParallelCluster Note:** Fast insufficient capacity fail-over is enabled by default (v3.2.0+). It disables the failing compute resource for `insufficient_capacity_timeout` (default 600s) and requeues jobs to other compute resources. Tune the timeout in `/etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf`.

> **PCS Note:** PCS manages capacity allocation automatically. If instances can't launch, check your compute node group's instance type configuration and consider adding alternative types.

---

## Bootstrap Failures

Compute nodes launch but fail to join the cluster. The instance starts but never registers with slurmctld.

### What You See

- Jobs stay `PENDING` indefinitely
- Nodes cycle through `POWERING_UP` → `DOWN` → `POWERING_UP`
- ParallelCluster enters **protected mode** after 10 consecutive failures

---

## Bootstrap Failure Types

| Type | Cause | Log to Check |
|------|-------|-------------|
| Instance self-terminates | Custom action (OnNodeConfigured) fails | `cloud-init-output.log` (compute node) |
| Resume timeout expires | Node can't reach slurmctld within timeout | `clustermgtd` (head node) |
| Health check failure | EC2 health check fails | `clustermgtd` (head node) |
| Invalid registration | Slurm config mismatch (CustomSlurmSettings error) | `slurmctld.log` (head node) |

---

## Bootstrap Failures: Diagnosis

```bash
# Check for protected mode (head node)
pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2
# Look for: "status": "PROTECTED"

# Check which nodes are failing (head node)
sinfo
# Partitions in INACTIVE state = protected mode triggered

# Check head node logs (head node)
sudo grep "bootstrap failure" /var/log/parallelcluster/clustermgtd
sudo grep "Node bootstrap error" /var/log/parallelcluster/clustermgtd

# Check compute node logs — if the node is still running (compute node)
sudo tail -100 /var/log/cloud-init-output.log
```

---

## Accessing Compute Node Logs

When a compute node fails bootstrap and self-terminates, you often can't SSH to it. Multiple options exist for retrieving logs from failed or inaccessible compute nodes.

### SSM Session Manager (preferred for running nodes)

```bash
# Find the instance ID from clustermgtd or the EC2 console (head node)
sudo grep "Launching" /var/log/parallelcluster/slurm_resume.log | tail -5

# Start a session to the compute node (local workstation)
aws ssm start-session --target i-0abc123def456 --region us-east-2

# Once connected (compute node, via SSM)
sudo tail -200 /var/log/cloud-init-output.log
sudo tail -100 /var/log/slurmd.log
```

> SSM requires the `AmazonSSMManagedInstanceCore` policy on the compute node instance profile. ParallelCluster enables this by default.

---

## Accessing Compute Node Logs (continued)

### CloudWatch Logs (works even after node terminates)

ParallelCluster ships compute node logs to CloudWatch automatically. Logs are preserved even after the node terminates.

```bash
# List all log streams for the cluster (local workstation)
pcluster list-cluster-log-streams --cluster-name mycluster \
  --region us-east-2 --filters "Name=node-type,Values=Compute"

# Retrieve events from a specific compute node log stream (local workstation)
pcluster get-cluster-log-events --cluster-name mycluster \
  --region us-east-2 \
  --log-stream-name ip-10-0-1-42.i-0abc123def456.cloud-init-output

# Export all cluster logs to S3 for offline analysis (local workstation)
pcluster export-cluster-logs --cluster-name mycluster \
  --region us-east-2 --bucket my-debug-bucket
```

---

## Accessing Compute Node Logs (continued)

### EC2 Console Output (last resort for terminated nodes)

When CloudWatch has no logs (node died before the agent could ship them), retrieve the raw EC2 serial console output.

```bash
# ParallelCluster auto-captures console output on the head node (head node)
cat /var/log/parallelcluster/compute_console_output

# Or retrieve manually with AWS CLI if you have the instance ID (local workstation)
aws ec2 get-console-output --instance-id i-0abc123def456 \
  --region us-east-2 --output text
```

### pcluster ssh (quick access to running nodes)

```bash
# SSH through the head node to a specific compute node (local workstation)
pcluster ssh --cluster-name mycluster --region us-east-2 \
  -i ~/.ssh/mykey.pem

# From the head node, hop to a compute node (head node)
ssh queue1-dy-c-1-1
```

---

## Bootstrap Failures: Common Causes

**Network issues:**

- Security group doesn't allow Slurm ports (6817, 6818, 6819)
- Compute subnet has no route to head node (missing NAT gateway or VPC endpoint)
- DNS resolution failing for head node hostname

**Custom action failures:**

- Script in `OnNodeConfigured` exits with non-zero status
- Script tries to download packages but has no internet access
- Script timeout exceeded
- Script runs `apt` or `dnf` (installs, or an `environment-modules` to `lmod` swap): on a freshly launched node's cold root volume that is minutes, not seconds, so the node can miss `ResumeTimeout` and be requeued. See [Measuring and shrinking cold start](aws-parallelcluster.md#measuring-and-shrinking-cold-start)
- Test job submitted from an SSM Session Manager shell fails instantly with `FAILED` and exit code `0:53`: the session's working directory (under the SSM agent's snap path) does not exist on the compute node, so Slurm cannot start the job. Submit with `sbatch --chdir=/tmp ...`; the same code appears when `--output` points into a directory that does not exist on the node

**Configuration mismatch:**

- `CustomSlurmSettings` for compute nodes has syntax errors
- Node definition in slurm.conf doesn't match actual instance hardware

---

## Bootstrap Failures: Recovery

```bash
# Exit protected mode (local workstation or head node)
pcluster update-compute-fleet --cluster-name mycluster \
  --region us-east-2 --status START_REQUESTED

# Fix the root cause first, then restart
# Common: fix security groups, custom action scripts, or CustomSlurmSettings
```

> **PCS Note:** Bootstrap failures in PCS are often caused by missing `pcs:RegisterComputeNodeGroupInstance` permission on the instance profile, VPC endpoint misconfiguration, or security group rules blocking traffic. Check instance logs via `aws ec2 get-console-output` or Systems Manager.

---

## Protected Mode Deep Dive

### What Protected Mode Is

Protected mode is a ParallelCluster safety mechanism that **automatically disables a queue (partition) after repeated compute node bootstrap failures**. It prevents runaway EC2 costs from a launch-fail-terminate-relaunch loop.

Without protected mode, a misconfigured cluster would continuously launch instances that immediately fail, burning through your budget while no jobs ever run.

### How It Works

1. `clustermgtd` (the ParallelCluster management daemon on the head node) tracks bootstrap failures per queue
2. When consecutive failures reach the `protected_failure_count` threshold, that queue is set to `INACTIVE`
3. The compute fleet status changes to `PROTECTED`
4. No new instances are launched for affected queues until you intervene

---

## Protected Mode: The Failure Threshold

The default threshold is **10 consecutive bootstrap failures** per queue. This is configured in the `clustermgtd` configuration file on the head node.

```bash
# View the current threshold (head node)
sudo grep protected_failure_count \
  /etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf
# Default: protected_failure_count = 10
```

**Tuning the threshold:**

| Setting | Effect | When to Use |
|---------|--------|-------------|
| Lower (e.g., 3-5) | Triggers faster, fewer wasted instances | Production clusters with well-tested configs |
| Higher (e.g., 20-30) | More tolerant of transient failures | Clusters with known intermittent capacity issues |
| `-1` (disable) | Protected mode never activates | **Not recommended** — risk of runaway costs |

To change the threshold, edit the config file and restart `clustermgtd`, or set `protected_failure_count` via a `pcluster update-cluster` with custom settings.

---

## Protected Mode: Detecting It

### From the CLI (local workstation or head node)

```bash
# Check compute fleet status (local workstation)
pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2
```

```json
{
  "status": "PROTECTED",
  "lastStatusUpdatedTime": "2026-04-12T10:23:45.000Z"
}
```

### From Slurm (head node)

```bash
# Affected partitions show as INACTIVE
sinfo
```

```
PARTITION AVAIL  TIMELIMIT NODES STATE  NODELIST
queue1*   inact  infinite  15    idle~  queue1-dy-c-1-[1-15]
queue2    up     infinite  8     idle~  queue2-dy-g6-[1-8]
```

A partition showing `inact` (INACTIVE) while other partitions remain `up` is the telltale sign. Only queues whose failure count reached the threshold are disabled.

---

## Protected Mode: The 4 Bootstrap Failure Types

Each of these failure types increments the per-queue failure counter in `clustermgtd`:

**1. Instance Self-Terminates**

- The EC2 instance launches but shuts itself down before registering
- Usually caused by a failing `OnNodeConfigured` custom action script
- Debug: check `cloud-init-output.log` via CloudWatch or `compute_console_output` on head node

**2. Resume Timeout Expires**

- Node launches but doesn't register with slurmctld within `ResumeTimeout` (default 1800s)
- Causes: slow bootstrap, network issues, DNS failure, package download stalls
- Debug: `clustermgtd` log shows "Node replacement timeout" or "resume timeout"

**3. EC2 Health Check Failure**

- Instance passes launch but fails AWS EC2 status checks (system or instance check)
- Causes: hardware fault, kernel panic, corrupted AMI
- Debug: `clustermgtd` log shows "Health check" failures; EC2 console shows impaired status

**4. Invalid Slurm Registration**

- Node contacts slurmctld but registration is rejected
- Causes: `CustomSlurmSettings` syntax errors, CPU/memory mismatch between config and actual hardware
- Debug: `slurmctld.log` on head node shows registration rejection messages

---

## Protected Mode: Debugging

### Step-by-step diagnosis (all from head node unless noted)

```bash
# 1. Confirm protected mode (head node)
pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2

# 2. Identify which queue(s) are affected (head node)
sinfo
# Look for partitions in INACTIVE state

# 3. Check clustermgtd for the failure pattern (head node)
sudo grep -E "(bootstrap failure|Node bootstrap error|protected mode)" \
  /var/log/parallelcluster/clustermgtd | tail -20

# 4. Identify the failing node and instance ID (head node)
sudo grep -E "(self-terminate|resume timeout|health check|registration)" \
  /var/log/parallelcluster/clustermgtd | tail -10

# 5. Get compute node logs (head node)
# Option A: auto-captured console output
cat /var/log/parallelcluster/compute_console_output

# Option B: CloudWatch (local workstation)
pcluster get-cluster-log-events --cluster-name mycluster \
  --region us-east-2 \
  --log-stream-name <stream-from-list-cluster-log-streams>

# Option C: EC2 console output (local workstation)
aws ec2 get-console-output --instance-id <id> --region us-east-2 --output text
```

---

## Protected Mode: Recovery

**Always fix the root cause before re-enabling the fleet.** Otherwise the cluster will just re-enter protected mode after another 10 failures.

### Re-enable the compute fleet

```bash
# Re-enable all queues (local workstation or head node)
pcluster update-compute-fleet --cluster-name mycluster \
  --region us-east-2 --status START_REQUESTED
```

This resets the failure counters and sets all queues back to `UP`. Verify recovery:

```bash
# Confirm fleet is running again (head node)
sinfo
# All partitions should show 'up' in AVAIL column

# Submit a small test job to verify nodes bootstrap successfully (head node)
srun --partition=queue1 -N 1 hostname
```

### Common root-cause fixes

| Root Cause | Fix |
|-----------|-----|
| `OnNodeConfigured` script fails | Fix the script, test locally, re-deploy via `pcluster update-cluster` |
| Security group blocks Slurm ports | Add inbound rules for TCP 6817-6819 from compute subnet |
| No internet access for package install | Add NAT gateway or S3/VPC endpoints to private subnets |
| `CustomSlurmSettings` syntax error | Validate slurm.conf syntax with `slurmd -C`, fix in cluster config |
| AMI incompatibility | Rebuild custom AMI from the correct ParallelCluster base AMI |

---

## Protected Mode: Tuning and Disabling

You can adjust protected mode behavior in the `clustermgtd` config.

```bash
# Edit the clustermgtd config (head node)
sudo vi /etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf
```

Key parameters:

```ini
# Number of consecutive failures before protecting a queue (default: 10)
protected_failure_count = 10

# Set to -1 to disable protected mode entirely (NOT recommended)
# protected_failure_count = -1
```

> **Warning:** Disabling protected mode (`-1`) removes the safety net against runaway instance launches. If you disable it, implement your own monitoring — for example, a CloudWatch alarm on `InstanceBootstrapTimeoutError` metric or a budget alert on EC2 spend.

After editing, restart `clustermgtd`:

```bash
# Restart the daemon to apply changes (head node)
sudo systemctl restart supervisord
```

> **ParallelCluster Note:** Changes to `parallelcluster_clustermgtd.conf` are not persisted across `pcluster update-cluster` operations. To make the setting permanent, use a `OnNodeConfigured` custom action on the head node to write the config value.

---

## Spot Instance Interruptions

Spot instances can be reclaimed by AWS with 2 minutes notice. Jobs on reclaimed nodes are killed.

### What Happens

1. AWS issues a Spot interruption notice (2 min warning)
2. ParallelCluster detects the notice and drains the node
3. Running jobs are terminated (or requeued if `--requeue` was used)
4. The instance is terminated
5. A replacement node is launched (if capacity is available)

---

## Spot Interruptions: Mitigation

**Job-level:**

```bash
# Enable automatic requeue on node failure
sbatch --requeue my_job.sh

# Or in the job script
#SBATCH --requeue
```

**Cluster-level:**

- Use On-Demand instances for long-running or non-restartable jobs
- Use Spot for short, restartable, or checkpointable workloads
- Mix On-Demand and Spot compute resources in the same queue
- Implement application-level checkpointing for long simulations

> **ParallelCluster Note:** Configure separate compute resources for Spot and On-Demand in the same queue. Users can target specific compute resources via Slurm constraints.

> **PCS Note:** Set the purchase option (On-Demand vs Spot) at the compute node group level. Consider separate queues for Spot and On-Demand workloads.

---

## Scaling Issues

Dynamic nodes that won't start, take too long, or scale down unexpectedly.

### Nodes Won't Resume from Power Saving

```bash
# Check node states — idle~ means power saving, should resume on job submit (head node)
sinfo -N -l | grep "idle~"

# If nodes stay in idle~ after job submission, check resume log (head node)
sudo tail -50 /var/log/parallelcluster/slurm_resume.log

# Check for EC2 launch failures (head node)
sudo grep "ERROR" /var/log/parallelcluster/slurm_resume.log
```

---

## Scaling Issues: Common Problems

| Symptom | Likely Cause | Fix |
|---------|-------------|-----|
| Nodes stuck in `POWERING_UP` | Bootstrap taking too long (often `apt`/`dnf` in `OnNodeConfigured`) | Shorten the boot: [cold start](aws-parallelcluster.md#measuring-and-shrinking-cold-start). `ResumeTimeout` is deny-listed in `CustomSlurmSettings` |
| Node `down` with `Node unexpectedly rebooted.`, job requeued after a long wait | Node missed `ResumeTimeout`; the requeue looks like a job failure | Same: find the slow phase, then bake it into the AMI |
| Job requeued after a resume failure although it asked for no requeue | `requeue_on_resume_failure` in `SchedulerParameters` | Remove it from `CustomSlurmSettings: - SchedulerParameters:` if unwanted |
| Submissions feel slow during a flood; clients sleep a second | `SlurmctldParameters=rl_enable` rate limiting (check `scontrol show config`) | Spread submissions or use job arrays; `SlurmctldParameters` is deny-listed |
| Nodes stuck in `COMPLETING` | Post-job cleanup stalled | Check epilog scripts, increase `EpilogMsgTime` |
| Nodes scale down too fast | Idle timeout too short | Increase `ScaledownIdletime` (it sets `SuspendTime`); `SuspendTimeout` only delays reuse of a powered-down node |
| Nodes scale down between pipeline stages | Gap between dependent jobs | Increase `ScaledownIdletime` to keep nodes warm |
| Max capacity silently reached | Max node count hit | Check `MaxCount` in cluster config, request quota increase |

Key-by-key reference: [Cloud Power-Save Keys](aws-parallelcluster.md#cloud-power-save-keys).

---

## Storage Failures

Shared filesystem issues that affect job execution on compute nodes.

### FSx for Lustre

```bash
# Check if Lustre is mounted (compute node)
mount | grep lustre

# If mount failed, check cloud-init and system logs (compute node)
dmesg | grep -i lustre
journalctl -u amazon-efs-mount-watchdog  # EFS
```

**Common issues:**

- FSx mount timeout during node bootstrap (security group blocking port 988)
- Lustre client version mismatch with kernel
- Throughput throttling under heavy I/O (check CloudWatch FSx metrics)

### EFS

- Bursting throughput exhausted — switch to Elastic throughput mode
- Mount target in wrong AZ — ensure mount targets exist in compute subnets

### Head Node Root Volume Full

- The pcluster head node has a small EBS root volume (50 GB by default) and no local ephemeral `/scratch` — on a head node instance without instance storage, there is no `/scratch` mount to begin with.
- Don't use the head node for job output. Redirect all job working dirs and outputs to shared storage (FSx Lustre, EFS, or an EBS mount at `/shared`).
- For per-compute-node scratch I/O, use `$SLURM_TMPDIR` or `/scratch` (on compute instance types that have instance storage); see `deployment/aws-parallelcluster.md` "Ephemeral Storage and /scratch".

---

## Network and DNS Issues

### Compute Nodes Can't Reach slurmctld

```bash
# Test connectivity to head node on Slurm port (compute node)
nc -zv <head-node-ip> 6817
nc -zv <head-node-ip> 6818

# Check security group allows inbound from compute subnet (AWS console or CLI)
# Check route tables have path between subnets
```

### DNS Resolution Failures

- ParallelCluster uses Route 53 private hosted zones for node name resolution
- If nodes can't resolve hostnames, check the VPC's DNS settings (`enableDnsHostnames`, `enableDnsSupport`)

> **PCS Note:** PCS manages DNS and networking for cluster communication. If nodes can't register, verify that the VPC endpoints for PCS are correctly configured and that the endpoint policy allows `pcs:RegisterComputeNodeGroupInstance`.

---

## Key Log Files

| Log | Location | Node | What It Contains |
|-----|----------|------|-----------------|
| Resume log | `/var/log/parallelcluster/slurm_resume.log` | Head node | Instance launch attempts, capacity errors |
| Suspend log | `/var/log/parallelcluster/slurm_suspend.log` | Head node | Node power-down events |
| Cluster manager | `/var/log/parallelcluster/clustermgtd` | Head node | Protected mode, bootstrap failures, node health |
| Console output | `/var/log/parallelcluster/compute_console_output` | Head node | Auto-captured EC2 console output from failed compute nodes |
| slurmctld | `/var/log/slurmctld.log` | Head node | Scheduler decisions, node registration |
| cloud-init | `/var/log/cloud-init-output.log` | Compute node | Bootstrap sequence, custom action output |
| slurmd | `/var/log/slurmd.log` | Compute node | Compute daemon startup, job execution |

> **PCS Note:** PCS compute node logs are accessible via CloudWatch Logs or Systems Manager. Use `aws ssm start-session` to access running compute nodes directly.

---

## Quick Diagnostic Checklist

When something goes wrong on an AWS Slurm cluster:

1. **Check node states:** `sinfo -a -N -o '%N %T %E'` (head node) — what's down and why? (`sinfo -R` lists only nodes that carry a reason; a silently `NOT_RESPONDING` node has none, and `-a` includes hidden partitions.)
2. **Check fleet status:** `pcluster describe-compute-fleet` (head node or local workstation) — protected mode?
3. **Check resume log:** `/var/log/parallelcluster/slurm_resume.log` (head node) — launch failures?
4. **Check clustermgtd:** `/var/log/parallelcluster/clustermgtd` (head node) — bootstrap failures?
5. **Check compute node:** `cloud-init-output.log` (compute node via SSM/CloudWatch/console output) — what failed during setup?
6. **Check quotas:** Service Quotas console (AWS console) — vCPU limits hit?
7. **Check networking:** Security groups, route tables, DNS (AWS console) — can nodes communicate?
8. **Collect a support bundle:** `pcluster-diag` (ParallelCluster 3.16+, on the head node) gathers the logs above plus cluster config into one archive — run it before opening an AWS support case.

Exercises¶

Interpret capacity error node states

A cluster shows the following sinfo output. What happened, and what should you do?

PARTITION AVAIL TIMELIMIT NODES STATE  NODELIST
gpu*      up    infinite  1     down#  gpu-dy-p5-1
gpu*      up    infinite  7     down~  gpu-dy-p5-[2-8]
gpu*      up    infinite  8     idle~  gpu-dy-g6-[1-8]

Hint / Solution

Node `gpu-dy-p5-1` failed to launch (likely `InsufficientInstanceCapacity` for p5.48xlarge). The `down#` state means it was the node that tried to launch. The `down~` nodes are other p5 nodes in the same compute resource, pre-emptively disabled by fast insufficient capacity fail-over. The `gpu-dy-g6` nodes are a different compute resource (g6 instances) and are still `idle~` (available). The job should automatically requeue to the g6 compute resource. **To verify:** `scontrol show nodes gpu-dy-p5-1 | grep Reason` **If the job didn't requeue:** Check that both compute resources are in the same queue and that the job's constraints allow g6 instances. Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sinfo -R
REASON               USER      TIMESTAMP           NODELIST

$ scontrol show node batch-dy-compute-1 | grep -E '(Reason|State)='
   State=IDLE+CLOUD ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A

Diagnose a bootstrap failure

Your cluster has entered protected mode. No jobs are running. How do you find out what went wrong and recover?

Hint / Solution

# 1. Confirm protected mode
pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2
# Look for "status": "PROTECTED"

# 2. Find which queue is affected
sinfo
# Partitions showing INACTIVE = protected mode triggered

# 3. Find the failing nodes
sudo grep "Node bootstrap error" /var/log/parallelcluster/clustermgtd

# 4. Check the compute node logs (get IP from step 3)
# SSH or SSM to the compute node, then:
sudo tail -100 /var/log/cloud-init-output.log

# 5. Fix the root cause (custom action, security group, etc.)

# 6. Exit protected mode
pcluster update-compute-fleet --cluster-name mycluster \
  --region us-east-2 --status START_REQUESTED

Expected output: Validated 2026-10-19 on slurm-training-val (Slurm 25.11.8, ParallelCluster 3.16.1).

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      3  idle~ batch-dy-compute-[2-4]
batch*       up   infinite      1   idle batch-dy-compute-1
debug        up   infinite      2  idle~ debug-dy-small-[1-2]
gpu          up   infinite      1  idle~ gpu-dy-t4-1
highmem      up   infinite      1  idle~ highmem-dy-large-1

$ sudo grep 'Node bootstrap error' /var/log/parallelcluster/clustermgtd 2>/dev/null | tail -5 || echo '(no bootstrap errors found)'

$ sudo tail -3 /var/log/parallelcluster/clustermgtd
2026-10-19 01:14:41,42 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Performing node maintenance actions
2026-10-19 01:14:41,43 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Following nodes are currently in replacement: (x0) []
2026-10-19 01:14:41,43 - [slurm_plugin.clustermgtd:_terminate_orphaned_instances] - INFO - Checking for orphaned instance

Time a cold start

A dynamic node is powered down (idle~). Measure how long a job takes to start on it, and find where the time went using the head node's resume log.

Hint / Solution

# 1. Pick a powered-down node (head node)
sinfo -h -N -p <queue> -o '%N %T' | grep 'idle~' | head -1

# 2. Submit a trivial job pinned to it. --chdir=/tmp: an SSM session's
#    working directory does not exist on the node (instant FAILED, 0:53)
id=$(sbatch --parsable -p <queue> -w <node> --chdir=/tmp --wrap=hostname)

# 3. Cold-start time = Start minus Submit
sacct -j "$id" -X -o JobID,Submit,Start,Elapsed,NodeList   # needs accounting
scontrol show job "$id" | grep -E 'SubmitTime|StartTime'   # fallback, while the job is in memory

# 4. When did ParallelCluster start launching the node? (head node)
sudo grep "<node>" /var/log/parallelcluster/slurm_resume.log | head -3

The first `slurm_resume.log` line for the node is the end of phase 1. `Start` minus that timestamp is phases 2 to 5 (instance boot, cloud-init, cookbook and your custom actions, `slurmd` registration). Run it three times; to split phase 4 out, read the node's `chef-client` stream in CloudWatch ([Accessing Compute Node Logs](#accessing-compute-node-logs)).

References¶

ParallelCluster: Troubleshooting Scaling Issues
ParallelCluster: Slurm Cluster Protected Mode
ParallelCluster: Fast Insufficient Capacity Fail-Over
ParallelCluster: Retrieving and Preserving Logs
ParallelCluster: EC2 Console Output Logs
PCS: Troubleshooting Compute Node Bootstrap
PCS: General Troubleshooting
ParallelCluster: Multiple Queue Mode Guide (ResumeTimeout/SuspendTimeout defaults, power-save flow)
ParallelCluster: Slurm Configuration Customization (deny-listed parameters)
Slurm Troubleshooting (general) — generic Slurm troubleshooting module