AWS Troubleshooting

<!-- audience: L4 --> <!-- deployment: aws-parallelcluster, aws-pcs --> # AWS Slurm Troubleshooting --- ## How AWS Slurm Differs Running Slurm on AWS introduces failure modes that don't exist on-premise. Nodes are ephemeral, capacity is shared, and network connectivity is dynamic. This module covers AWS-specific issues. For general Slurm troubleshooting (pending jobs, node states, log files), see the [Troubleshooting](../admin/troubleshooting.md) module. ### Key Differences from On-Premise - Nodes can **fail to launch** (capacity, quotas, AMI issues) - Nodes can **disappear mid-job** (Spot interruptions, health checks) - Nodes must **bootstrap and register** before running jobs (cloud-init, network, DNS) - **Scaling adds latency** between job submission and execution --- ## Insufficient Capacity Errors The most common AWS-specific failure. EC2 cannot provide the requested instance type in the requested Availability Zone. ### What You See ``` $ sinfo PARTITION AVAIL TIMELIMIT NODES STATE NODELIST queue1* up infinite 1 down# queue1-dy-c-1-1 queue1* up infinite 14 down~ queue1-dy-c-1-[2-15] ``` Nodes in `down#` or `down~` with reason: ``` (Code:InsufficientInstanceCapacity)Failure when resuming nodes. ``` --- ## Insufficient Capacity: Error Codes | Error Code | Meaning | |-----------|---------| | `InsufficientInstanceCapacity` | No instances available in this AZ | | `InsufficientHostCapacity` | Dedicated Host capacity exhausted | | `MaxSpotInstanceCountExceeded` | Spot instance limit reached | | `SpotMaxPriceTooLow` | Spot bid below minimum fulfillment price | | `Unsupported` | Instance type not available in this Region/AZ | | `VcpuLimitExceeded` | Account vCPU service quota hit | --- ## Insufficient Capacity: Resolution **Immediate steps:** ```bash # Check which nodes are affected and why (head node) sinfo -R scontrol show nodes queue1-dy-c-1-1 | grep Reason # Check the resume log for details (head node) sudo tail -50 /var/log/parallelcluster/slurm_resume.log ``` **Fixes:** - **Add multiple instance types** to your compute resource — ParallelCluster will fail over to alternatives - **Use multiple AZs** via subnet configuration - **Request a service quota increase** for vCPUs (Service Quotas console) - **Use Capacity Reservations** for critical instance types > **ParallelCluster Note:** Fast insufficient capacity fail-over is enabled by default (v3.2.0+). It disables the failing compute resource for `insufficient_capacity_timeout` (default 600s) and requeues jobs to other compute resources. Tune the timeout in `/etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf`. > **PCS Note:** PCS manages capacity allocation automatically. If instances can't launch, check your compute node group's instance type configuration and consider adding alternative types. --- ## Bootstrap Failures Compute nodes launch but fail to join the cluster. The instance starts but never registers with slurmctld. ### What You See - Jobs stay `PENDING` indefinitely - Nodes cycle through `POWERING_UP` → `DOWN` → `POWERING_UP` - ParallelCluster enters **protected mode** after 10 consecutive failures --- ## Bootstrap Failure Types | Type | Cause | Log to Check | |------|-------|-------------| | Instance self-terminates | Custom action (OnNodeConfigured) fails | `cloud-init-output.log` (compute node) | | Resume timeout expires | Node can't reach slurmctld within timeout | `clustermgtd` (head node) | | Health check failure | EC2 health check fails | `clustermgtd` (head node) | | Invalid registration | Slurm config mismatch (CustomSlurmSettings error) | `slurmctld.log` (head node) | --- ## Bootstrap Failures: Diagnosis ```bash # Check for protected mode (head node) pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2 # Look for: "status": "PROTECTED" # Check which nodes are failing (head node) sinfo # Partitions in INACTIVE state = protected mode triggered # Check head node logs (head node) sudo grep "bootstrap failure" /var/log/parallelcluster/clustermgtd sudo grep "Node bootstrap error" /var/log/parallelcluster/clustermgtd # Check compute node logs — if the node is still running (compute node) sudo tail -100 /var/log/cloud-init-output.log ``` --- ## Accessing Compute Node Logs When a compute node fails bootstrap and self-terminates, you often can't SSH to it. Multiple options exist for retrieving logs from failed or inaccessible compute nodes. ### SSM Session Manager (preferred for running nodes) ```bash # Find the instance ID from clustermgtd or the EC2 console (head node) sudo grep "Launching" /var/log/parallelcluster/slurm_resume.log | tail -5 # Start a session to the compute node (local workstation) aws ssm start-session --target i-0abc123def456 --region us-east-2 # Once connected (compute node, via SSM) sudo tail -200 /var/log/cloud-init-output.log sudo tail -100 /var/log/slurmd.log ``` > SSM requires the `AmazonSSMManagedInstanceCore` policy on the compute node instance profile. ParallelCluster enables this by default. --- ## Accessing Compute Node Logs (continued) ### CloudWatch Logs (works even after node terminates) ParallelCluster ships compute node logs to CloudWatch automatically. Logs are preserved even after the node terminates. ```bash # List all log streams for the cluster (local workstation) pcluster list-cluster-log-streams --cluster-name mycluster \ --region us-east-2 --filters "Name=node-type,Values=Compute" # Retrieve events from a specific compute node log stream (local workstation) pcluster get-cluster-log-events --cluster-name mycluster \ --region us-east-2 \ --log-stream-name ip-10-0-1-42.i-0abc123def456.cloud-init-output # Export all cluster logs to S3 for offline analysis (local workstation) pcluster export-cluster-logs --cluster-name mycluster \ --region us-east-2 --bucket my-debug-bucket ``` --- ## Accessing Compute Node Logs (continued) ### EC2 Console Output (last resort for terminated nodes) When CloudWatch has no logs (node died before the agent could ship them), retrieve the raw EC2 serial console output. ```bash # ParallelCluster auto-captures console output on the head node (head node) cat /var/log/parallelcluster/compute_console_output # Or retrieve manually with AWS CLI if you have the instance ID (local workstation) aws ec2 get-console-output --instance-id i-0abc123def456 \ --region us-east-2 --output text ``` ### pcluster ssh (quick access to running nodes) ```bash # SSH through the head node to a specific compute node (local workstation) pcluster ssh --cluster-name mycluster --region us-east-2 \ -i ~/.ssh/mykey.pem # From the head node, hop to a compute node (head node) ssh queue1-dy-c-1-1 ``` --- ## Bootstrap Failures: Common Causes **Network issues:** - Security group doesn't allow Slurm ports (6817, 6818, 6819) - Compute subnet has no route to head node (missing NAT gateway or VPC endpoint) - DNS resolution failing for head node hostname **Custom action failures:** - Script in `OnNodeConfigured` exits with non-zero status - Script tries to download packages but has no internet access - Script timeout exceeded **Configuration mismatch:** - `CustomSlurmSettings` for compute nodes has syntax errors - Node definition in slurm.conf doesn't match actual instance hardware --- ## Bootstrap Failures: Recovery ```bash # Exit protected mode (local workstation or head node) pcluster update-compute-fleet --cluster-name mycluster \ --region us-east-2 --status START_REQUESTED # Fix the root cause first, then restart # Common: fix security groups, custom action scripts, or CustomSlurmSettings ``` > **PCS Note:** Bootstrap failures in PCS are often caused by missing `pcs:RegisterComputeNodeGroupInstance` permission on the instance profile, VPC endpoint misconfiguration, or security group rules blocking traffic. Check instance logs via `aws ec2 get-console-output` or Systems Manager. --- ## Protected Mode Deep Dive ### What Protected Mode Is Protected mode is a ParallelCluster safety mechanism that **automatically disables a queue (partition) after repeated compute node bootstrap failures**. It prevents runaway EC2 costs from a launch-fail-terminate-relaunch loop. Without protected mode, a misconfigured cluster would continuously launch instances that immediately fail, burning through your budget while no jobs ever run. ### How It Works 1. `clustermgtd` (the ParallelCluster management daemon on the head node) tracks bootstrap failures per queue 2. When consecutive failures reach the `protected_failure_count` threshold, that queue is set to `INACTIVE` 3. The compute fleet status changes to `PROTECTED` 4. No new instances are launched for affected queues until you intervene --- ## Protected Mode: The Failure Threshold The default threshold is **10 consecutive bootstrap failures** per queue. This is configured in the `clustermgtd` configuration file on the head node. ```bash # View the current threshold (head node) sudo grep protected_failure_count \ /etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf # Default: protected_failure_count = 10 ``` **Tuning the threshold:** | Setting | Effect | When to Use | |---------|--------|-------------| | Lower (e.g., 3-5) | Triggers faster, fewer wasted instances | Production clusters with well-tested configs | | Higher (e.g., 20-30) | More tolerant of transient failures | Clusters with known intermittent capacity issues | | `-1` (disable) | Protected mode never activates | **Not recommended** — risk of runaway costs | To change the threshold, edit the config file and restart `clustermgtd`, or set `protected_failure_count` via a `pcluster update-cluster` with custom settings. --- ## Protected Mode: Detecting It ### From the CLI (local workstation or head node) ```bash # Check compute fleet status (local workstation) pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2 ``` ```json { "status": "PROTECTED", "lastStatusUpdatedTime": "2026-04-12T10:23:45.000Z" } ``` ### From Slurm (head node) ```bash # Affected partitions show as INACTIVE sinfo ``` ``` PARTITION AVAIL TIMELIMIT NODES STATE NODELIST queue1* inact infinite 15 idle~ queue1-dy-c-1-[1-15] queue2 up infinite 8 idle~ queue2-dy-g5-[1-8] ``` A partition showing `inact` (INACTIVE) while other partitions remain `up` is the telltale sign. Only queues whose failure count reached the threshold are disabled. --- ## Protected Mode: The 4 Bootstrap Failure Types Each of these failure types increments the per-queue failure counter in `clustermgtd`: **1. Instance Self-Terminates** - The EC2 instance launches but shuts itself down before registering - Usually caused by a failing `OnNodeConfigured` custom action script - Debug: check `cloud-init-output.log` via CloudWatch or `compute_console_output` on head node **2. Resume Timeout Expires** - Node launches but doesn't register with slurmctld within `ResumeTimeout` (default 1800s) - Causes: slow bootstrap, network issues, DNS failure, package download stalls - Debug: `clustermgtd` log shows "Node replacement timeout" or "resume timeout" **3. EC2 Health Check Failure** - Instance passes launch but fails AWS EC2 status checks (system or instance check) - Causes: hardware fault, kernel panic, corrupted AMI - Debug: `clustermgtd` log shows "Health check" failures; EC2 console shows impaired status **4. Invalid Slurm Registration** - Node contacts slurmctld but registration is rejected - Causes: `CustomSlurmSettings` syntax errors, CPU/memory mismatch between config and actual hardware - Debug: `slurmctld.log` on head node shows registration rejection messages --- ## Protected Mode: Debugging ### Step-by-step diagnosis (all from head node unless noted) ```bash # 1. Confirm protected mode (head node) pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2 # 2. Identify which queue(s) are affected (head node) sinfo # Look for partitions in INACTIVE state # 3. Check clustermgtd for the failure pattern (head node) sudo grep -E "(bootstrap failure|Node bootstrap error|protected mode)" \ /var/log/parallelcluster/clustermgtd | tail -20 # 4. Identify the failing node and instance ID (head node) sudo grep -E "(self-terminate|resume timeout|health check|registration)" \ /var/log/parallelcluster/clustermgtd | tail -10 # 5. Get compute node logs (head node) # Option A: auto-captured console output cat /var/log/parallelcluster/compute_console_output # Option B: CloudWatch (local workstation) pcluster get-cluster-log-events --cluster-name mycluster \ --region us-east-2 \ --log-stream-name <stream-from-list-cluster-log-streams> # Option C: EC2 console output (local workstation) aws ec2 get-console-output --instance-id <id> --region us-east-2 --output text ``` --- ## Protected Mode: Recovery **Always fix the root cause before re-enabling the fleet.** Otherwise the cluster will just re-enter protected mode after another 10 failures. ### Re-enable the compute fleet ```bash # Re-enable all queues (local workstation or head node) pcluster update-compute-fleet --cluster-name mycluster \ --region us-east-2 --status START_REQUESTED ``` This resets the failure counters and sets all queues back to `UP`. Verify recovery: ```bash # Confirm fleet is running again (head node) sinfo # All partitions should show 'up' in AVAIL column # Submit a small test job to verify nodes bootstrap successfully (head node) srun --partition=queue1 -N 1 hostname ``` ### Common root-cause fixes | Root Cause | Fix | |-----------|-----| | `OnNodeConfigured` script fails | Fix the script, test locally, re-deploy via `pcluster update-cluster` | | Security group blocks Slurm ports | Add inbound rules for TCP 6817-6819 from compute subnet | | No internet access for package install | Add NAT gateway or S3/VPC endpoints to private subnets | | `CustomSlurmSettings` syntax error | Validate slurm.conf syntax with `slurmd -C`, fix in cluster config | | AMI incompatibility | Rebuild custom AMI from the correct ParallelCluster base AMI | --- ## Protected Mode: Tuning and Disabling You can adjust protected mode behavior in the `clustermgtd` config. ```bash # Edit the clustermgtd config (head node) sudo vi /etc/parallelcluster/slurm_plugin/parallelcluster_clustermgtd.conf ``` Key parameters: ```ini # Number of consecutive failures before protecting a queue (default: 10) protected_failure_count = 10 # Set to -1 to disable protected mode entirely (NOT recommended) # protected_failure_count = -1 ``` > **Warning:** Disabling protected mode (`-1`) removes the safety net against runaway instance launches. If you disable it, implement your own monitoring — for example, a CloudWatch alarm on `InstanceBootstrapTimeoutError` metric or a budget alert on EC2 spend. After editing, restart `clustermgtd`: ```bash # Restart the daemon to apply changes (head node) sudo systemctl restart supervisord ``` > **ParallelCluster Note:** Changes to `parallelcluster_clustermgtd.conf` are not persisted across `pcluster update-cluster` operations. To make the setting permanent, use a `OnNodeConfigured` custom action on the head node to write the config value. --- ## Spot Instance Interruptions Spot instances can be reclaimed by AWS with 2 minutes notice. Jobs on reclaimed nodes are killed. ### What Happens 1. AWS issues a Spot interruption notice (2 min warning) 2. ParallelCluster detects the notice and drains the node 3. Running jobs are terminated (or requeued if `--requeue` was used) 4. The instance is terminated 5. A replacement node is launched (if capacity is available) --- ## Spot Interruptions: Mitigation **Job-level:** ```bash # Enable automatic requeue on node failure sbatch --requeue my_job.sh # Or in the job script #SBATCH --requeue ``` **Cluster-level:** - Use On-Demand instances for long-running or non-restartable jobs - Use Spot for short, restartable, or checkpointable workloads - Mix On-Demand and Spot compute resources in the same queue - Implement application-level checkpointing for long simulations > **ParallelCluster Note:** Configure separate compute resources for Spot and On-Demand in the same queue. Users can target specific compute resources via Slurm constraints. > **PCS Note:** Set the purchase option (On-Demand vs Spot) at the compute node group level. Consider separate queues for Spot and On-Demand workloads. --- ## Scaling Issues Dynamic nodes that won't start, take too long, or scale down unexpectedly. ### Nodes Won't Resume from Power Saving ```bash # Check node states — idle~ means power saving, should resume on job submit (head node) sinfo -N -l | grep "idle~" # If nodes stay in idle~ after job submission, check resume log (head node) sudo tail -50 /var/log/parallelcluster/slurm_resume.log # Check for EC2 launch failures (head node) sudo grep "ERROR" /var/log/parallelcluster/slurm_resume.log ``` --- ## Scaling Issues: Common Problems | Symptom | Likely Cause | Fix | |---------|-------------|-----| | Nodes stuck in `POWERING_UP` | Bootstrap taking too long | Increase `ResumeTimeout` in CustomSlurmSettings | | Nodes stuck in `COMPLETING` | Post-job cleanup stalled | Check epilog scripts, increase `EpilogMsgTime` | | Nodes scale down too fast | Idle timeout too short | Increase `SuspendTimeout` or `ScaledownIdletime` | | Nodes scale down between pipeline stages | Gap between dependent jobs | Increase `SuspendTimeout` to keep nodes warm | | Max capacity silently reached | Max node count hit | Check `MaxCount` in cluster config, request quota increase | --- ## Storage Failures Shared filesystem issues that affect job execution on compute nodes. ### FSx for Lustre ```bash # Check if Lustre is mounted (compute node) mount | grep lustre # If mount failed, check cloud-init and system logs (compute node) dmesg | grep -i lustre journalctl -u amazon-efs-mount-watchdog # EFS ``` **Common issues:** - FSx mount timeout during node bootstrap (security group blocking port 988) - Lustre client version mismatch with kernel - Throughput throttling under heavy I/O (check CloudWatch FSx metrics) ### EFS - Bursting throughput exhausted — switch to Elastic throughput mode - Mount target in wrong AZ — ensure mount targets exist in compute subnets ### Head Node Root Volume Full - The pcluster head node has a small EBS root volume (50 GB by default) and no local ephemeral `/scratch` — on a head node instance without instance storage, there is no `/scratch` mount to begin with. - Don't use the head node for job output. Redirect all job working dirs and outputs to shared storage (FSx Lustre, EFS, or an EBS mount at `/shared`). - For per-compute-node scratch I/O, use `$SLURM_TMPDIR` or `/scratch` (on compute instance types that have instance storage); see `deployment/aws-parallelcluster.md` "Ephemeral Storage and /scratch". --- ## Network and DNS Issues ### Compute Nodes Can't Reach slurmctld ```bash # Test connectivity to head node on Slurm port (compute node) nc -zv <head-node-ip> 6817 nc -zv <head-node-ip> 6818 # Check security group allows inbound from compute subnet (AWS console or CLI) # Check route tables have path between subnets ``` ### DNS Resolution Failures - ParallelCluster uses Route 53 private hosted zones for node name resolution - If nodes can't resolve hostnames, check the VPC's DNS settings (`enableDnsHostnames`, `enableDnsSupport`) > **PCS Note:** PCS manages DNS and networking for cluster communication. If nodes can't register, verify that the VPC endpoints for PCS are correctly configured and that the endpoint policy allows `pcs:RegisterComputeNodeGroupInstance`. --- ## Key Log Files | Log | Location | Node | What It Contains | |-----|----------|------|-----------------| | Resume log | `/var/log/parallelcluster/slurm_resume.log` | Head node | Instance launch attempts, capacity errors | | Suspend log | `/var/log/parallelcluster/slurm_suspend.log` | Head node | Node power-down events | | Cluster manager | `/var/log/parallelcluster/clustermgtd` | Head node | Protected mode, bootstrap failures, node health | | Console output | `/var/log/parallelcluster/compute_console_output` | Head node | Auto-captured EC2 console output from failed compute nodes | | slurmctld | `/var/log/slurmctld.log` | Head node | Scheduler decisions, node registration | | cloud-init | `/var/log/cloud-init-output.log` | Compute node | Bootstrap sequence, custom action output | | slurmd | `/var/log/slurmd.log` | Compute node | Compute daemon startup, job execution | > **PCS Note:** PCS compute node logs are accessible via CloudWatch Logs or Systems Manager. Use `aws ssm start-session` to access running compute nodes directly. --- ## Quick Diagnostic Checklist When something goes wrong on an AWS Slurm cluster: 1. **Check node states:** `sinfo -R` (head node) — what's down and why? 2. **Check fleet status:** `pcluster describe-compute-fleet` (head node or local workstation) — protected mode? 3. **Check resume log:** `/var/log/parallelcluster/slurm_resume.log` (head node) — launch failures? 4. **Check clustermgtd:** `/var/log/parallelcluster/clustermgtd` (head node) — bootstrap failures? 5. **Check compute node:** `cloud-init-output.log` (compute node via SSM/CloudWatch/console output) — what failed during setup? 6. **Check quotas:** Service Quotas console (AWS console) — vCPU limits hit? 7. **Check networking:** Security groups, route tables, DNS (AWS console) — can nodes communicate?

Exercises

1. Interpret capacity error node states

A cluster shows the following sinfo output. What happened, and what should you do?

PARTITION AVAIL TIMELIMIT NODES STATE  NODELIST
gpu*      up    infinite  1     down#  gpu-dy-p4d-1
gpu*      up    infinite  7     down~  gpu-dy-p4d-[2-8]
gpu*      up    infinite  8     idle~  gpu-dy-g5-[1-8]

Hint / Solution

Node `gpu-dy-p4d-1` failed to launch (likely `InsufficientInstanceCapacity` for p4d.24xlarge). The `down#` state means it was the node that tried to launch. The `down~` nodes are other p4d nodes in the same compute resource, pre-emptively disabled by fast insufficient capacity fail-over. The `gpu-dy-g5` nodes are a different compute resource (g5 instances) and are still `idle~` (available). The job should automatically requeue to the g5 compute resource. **To verify:** `scontrol show nodes gpu-dy-p4d-1 | grep Reason` **If the job didn't requeue:** Check that both compute resources are in the same queue and that the job's constraints allow g5 instances. Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sinfo -R
REASON               USER      TIMESTAMP           NODELIST

$ scontrol show node batch-dy-compute-1 | grep -E '(Reason|State)='
   State=IDLE+CLOUD ThreadsPerCore=1 TmpDisk=0 Weight=1000 Owner=N/A MCS_label=N/A

1. Diagnose a bootstrap failure

Your cluster has entered protected mode. No jobs are running. How do you find out what went wrong and recover?

Hint / Solution

# 1. Confirm protected mode
pcluster describe-compute-fleet --cluster-name mycluster --region us-east-2
# Look for "status": "PROTECTED"

# 2. Find which queue is affected
sinfo
# Partitions showing INACTIVE = protected mode triggered

# 3. Find the failing nodes
sudo grep "Node bootstrap error" /var/log/parallelcluster/clustermgtd

# 4. Check the compute node logs (get IP from step 3)
# SSH or SSM to the compute node, then:
sudo tail -100 /var/log/cloud-init-output.log

# 5. Fix the root cause (custom action, security group, etc.)

# 6. Exit protected mode
pcluster update-compute-fleet --cluster-name mycluster \
  --region us-east-2 --status START_REQUESTED

Expected output: Validated 2026-04-15 on slurm-training-val (Slurm 25.11.4, ParallelCluster 3.15.0).

$ sinfo
PARTITION AVAIL  TIMELIMIT  NODES  STATE NODELIST
batch*       up   infinite      2  idle~ batch-dy-compute-[3-4]
batch*       up   infinite      2   idle batch-dy-compute-[1-2]
debug        up   infinite      2  idle~ debug-dy-small-[1-2]
gpu          up   infinite      1  idle~ gpu-dy-t4-1
highmem      up   infinite      1  idle~ highmem-dy-large-1

$ sudo grep 'Node bootstrap error' /var/log/parallelcluster/clustermgtd 2>/dev/null | tail -5 || echo '(no bootstrap errors found)'
2026-04-14 22:10:56,42 - [slurm_plugin.slurm_resources:is_bootstrap_failure] - WARNING - Node bootstrap error: Node gpu-dy-t4-1(172.31.89.235) failed to register to the Slurm management daemon, node state: IDLE+CLOUD+DRAIN+INVALID_REG
2026-04-14 23:25:44,43 - [slurm_plugin.slurm_resources:is_bootstrap_failure] - WARNING - Node bootstrap error: Node gpu-dy-t4-1(172.31.87.111) failed to register to the Slurm management daemon, node state: IDLE+CLOUD+DRAIN+INVALID_REG
2026-04-15 00:12:44,44 - [slurm_plugin.slurm_resources:is_bootstrap_failure] - WARNING - Node bootstrap error: Node gpu-dy-t4-1(172.31.80.161) failed to register to the Slurm management daemon, node state: IDLE+CLOUD+DRAIN+INVALID_REG

$ sudo tail -3 /var/log/parallelcluster/clustermgtd
2026-04-15 02:06:44,45 - [slurm_plugin.clustermgtd:_maintain_nodes] - INFO - Found the following unhealthy dynamic nodes: (x1) ['debug-dy-small-2(debug-dy-small-2)']
2026-04-15 02:06:44,45 - [slurm_plugin.clustermgtd:_handle_unhealthy_dynamic_nodes] - INFO - Setting unhealthy dynamic nodes to down and power_down.
2026-04-15 02:06:44,46 - [slurm_plugin.clustermgtd:_terminate_orphaned_instances] - INFO - Checking for orphaned instance

References

• ParallelCluster: Troubleshooting Scaling Issues
• ParallelCluster: Slurm Cluster Protected Mode
• ParallelCluster: Fast Insufficient Capacity Fail-Over
• ParallelCluster: Retrieving and Preserving Logs
• ParallelCluster: EC2 Console Output Logs
• PCS: Troubleshooting Compute Node Bootstrap
• PCS: General Troubleshooting
• Slurm Troubleshooting (general) — generic Slurm troubleshooting module