CryoSPARC User

# CryoSPARC & Slurm -- End-User Guide

## Overview

CryoSPARC is a GPU-accelerated platform for cryo-EM single particle analysis (SPA). It provides a web-based interface for the entire processing pipeline: motion correction, CTF estimation, particle picking, 2D classification, ab-initio reconstruction, 3D refinement, and more.

When CryoSPARC is deployed on a Slurm cluster, you interact with the CryoSPARC web UI to set up and run processing jobs. Behind the scenes, CryoSPARC translates your requests into `sbatch` submissions to Slurm. You do not need to write Slurm job scripts -- CryoSPARC handles that automatically.

### How It Works

```
You (CryoSPARC Web UI)
    |
    v
CryoSPARC Master (scheduler)
    |  Renders cluster_script.sh template with your resource selections
    |  Submits via sbatch, monitors via squeue, cancels via scancel
    v
Slurm (sbatch)
    |  Allocates GPUs, CPUs, memory on cluster nodes
    v
Compute Node (runs cryosparc_worker process on allocated GPUs)
```

The key concept: your admin has configured **lanes** in CryoSPARC that correspond to Slurm cluster partitions (or specific node groups). Each lane defines how CryoSPARC submits jobs to Slurm -- the partition, GPU type, and any site-specific resource limits. You select a lane when you queue a job, and CryoSPARC generates and submits the appropriate Slurm batch script.

### Lanes Are Like Slurm Partitions

A CryoSPARC lane maps to a cluster configuration. Common setups include:

| CryoSPARC Lane | Typical Slurm Mapping | Use Case |
|----------------|----------------------|----------|
| `gpu-a100` | `--partition=gpu --gres=gpu:a100:N` | High-end refinement, large box sizes |
| `gpu-v100` | `--partition=gpu --gres=gpu:v100:N` | General processing |
| `gpu-rtx3090` | `--partition=gpu --gres=gpu:rtx3090:N` | Cost-effective consumer GPUs |
| `workstation` | Local (no Slurm) | Quick tests on the master node |

Your admin may have different lane names. The lanes available to you appear in the Queue panel when you submit a job.

### What's Inside a Lane

Each lane is defined by two files that your admin has configured:

- **`cluster_info.json`** -- defines the Slurm commands (sbatch, squeue, scancel) and paths
- **`cluster_script.sh`** -- a Jinja2 template containing `#SBATCH` directives that CryoSPARC fills in with your job's resource requirements

Here is a simplified example of what these files look like for an `slurm-a100` lane:

**cluster_info.json** (defines the lane name and Slurm command paths):
```json
{
    "name": "slurm-a100",
    "worker_bin_path": "/shared/cryosparc_worker/bin/cryosparcw",
    "cache_path": "/scratch/cryosparc_cache",
    "qsub_cmd_tpl": "/usr/bin/sbatch {{ script_path_abs }}",
    "qstat_cmd_tpl": "/usr/bin/squeue -j {{ cluster_job_id }}",
    "qdel_cmd_tpl": "/usr/bin/scancel {{ cluster_job_id }}",
    "qinfo_cmd_tpl": "/usr/bin/sinfo"
}
```

**cluster_script.sh** (the Slurm submission template -- this is where the partition and GPU type are locked in):
```bash
#!/usr/bin/env bash
#SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }}
#SBATCH --partition=gpu-a100
#SBATCH --cpus-per-task={{ num_cpu }}
#SBATCH --gres=gpu:a100:{{ num_gpu }}
#SBATCH --mem={{ ram_gb|int }}G
#SBATCH --output={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.out
#SBATCH --error={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.err

{{ run_cmd }}
```

When you select the `slurm-a100` lane and queue a job requesting 2 GPUs, CryoSPARC fills in the template variables (`{{ num_gpu }}` becomes `2`, `{{ num_cpu }}` becomes the estimated CPU count, etc.) and submits the rendered script via `sbatch`. This is why choosing the right lane matters -- the `--partition` and `--gres=gpu:a100` are baked into the lane's template.

### Viewing Lane Configuration

You can see which lanes are registered and inspect their scripts:

```bash
# List all registered cluster lanes (requires shell access)
cryosparcm cli "get_scheduler_lanes()"

# Dump a lane's cluster_info.json and cluster_script.sh to the current directory
cryosparcm cluster dump slurm-a100
cat cluster_info.json    # See the lane name and Slurm command templates
cat cluster_script.sh    # See the #SBATCH directives (partition, GPU type, etc.)
```

You can also view the **rendered** (filled-in) submission script after a job has been submitted by looking in the job directory:

```bash
# Find the rendered script for a completed or running job
ls /path/to/project/J42/
# ... cluster_job.sh (the actual sbatch script that was submitted)
# ... P3_J42_slurm.out (Slurm stdout)
# ... P3_J42_slurm.err (Slurm stderr)

cat /path/to/project/J42/cluster_job.sh
# Shows the actual #SBATCH lines with all variables filled in
```

This is useful for understanding exactly what resources Slurm was asked for when troubleshooting failures.

> **ParallelCluster Note:** On AWS ParallelCluster, lanes typically map to compute resource groups with specific instance types (e.g., `p4d.24xlarge` with A100 GPUs or `g5.xlarge` with A10G GPUs). The `--partition` in the cluster script targets the ParallelCluster queue name (e.g., `gpu`). Nodes are provisioned on demand and may take 2-5 minutes to spin up.

> **PCS Note:** AWS PCS clusters work similarly to ParallelCluster. Your admin will have configured CryoSPARC lanes to target the appropriate PCS queue names and GPU instance types.

---

## Selecting Resources in CryoSPARC

When you build a job in CryoSPARC and click **Queue Job**, the Queue panel presents several resource settings. These are translated directly into Slurm `#SBATCH` directives.

### Choosing a Lane

1. Click **Queue Job** on the job builder
2. In the Queue slide-over, select the lane from the dropdown
3. CryoSPARC v5.0+ shows resource utilization for each lane, warning you if resources are fully allocated

Choose a lane based on the GPU type and availability. If your cluster has both A100 and V100 GPUs in separate partitions, selecting the right lane determines which GPU type your job gets.

### GPU Count

The number of GPUs is set in the job builder under **Compute Settings** at the bottom of the parameter list. Different job types use GPUs differently:

| Job Type | GPU Scaling | Recommendation |
|----------|-------------|----------------|
| Patch Motion Correction | Multi-GPU | 2-4 GPUs for throughput |
| Patch CTF Estimation | Multi-GPU | 2-4 GPUs for throughput |
| 2D Classification | Multi-GPU | 1-2 GPUs usually sufficient |
| Ab-Initio Reconstruction | Single GPU per class | 1 GPU per class (e.g., 3 classes = 3 GPUs) |
| Homogeneous Refinement | Single GPU | 1 GPU |
| Non-Uniform Refinement | Single GPU | 1 GPU |
| 3D Variability | Single GPU | 1 GPU |
| Local Refinement | Single GPU | 1 GPU |

Jobs that only use a single GPU will not benefit from requesting more GPUs. Requesting excess GPUs wastes cluster resources and may increase your queue wait time.

### CPU and Memory Settings

CryoSPARC estimates the CPU and memory requirements for each job type and passes these to Slurm via the `{{ num_cpu }}` and `{{ ram_gb }}` template variables. The resulting Slurm directives look like:

```bash
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --gres=gpu:2
```

In most cases, the defaults are appropriate. If your admin has configured custom variables in the cluster script template, you may see additional fields in the Queue panel (for example, a RAM multiplier or a specific partition override).

CryoSPARC's recommended minimum is 4 CPU cores and 64 GB RAM per GPU on worker nodes. If you find jobs running slowly, check with your admin whether the cluster script is requesting sufficient resources from Slurm.

### Run on Specific GPUs

For workstation (non-cluster) lanes, CryoSPARC offers a **Run on specific GPUs** tab in the Queue panel that lets you select individual GPU devices by index. This is useful for:

- Testing on a specific GPU when troubleshooting hardware issues
- Avoiding a GPU that another process is using
- Running benchmarks on a particular device

On cluster lanes, GPU assignment is handled by Slurm's GRES system, so this option is generally not available -- Slurm decides which physical GPUs your job gets.

---

## Monitoring Jobs

### CryoSPARC Job Status

In the CryoSPARC web UI, each job card shows its current status with a color indicator:

| CryoSPARC Status | Meaning |
|-----------------|---------|
| **Building** (grey) | Job is being configured, not yet submitted |
| **Queued** (blue) | Submitted to Slurm, waiting for resources |
| **Running** (green) | Executing on a compute node |
| **Completed** (green check) | Finished successfully |
| **Failed** (red) | Terminated with an error |
| **Killed** (orange) | Manually stopped by user or admin |

When a job shows **Queued**, it has been submitted to Slurm via `sbatch` and is waiting in the Slurm queue for resources to become available.

### Viewing the Event Log

Select a job card and press **Spacebar** (or click the job to open the inspector) to see the Event Log. This log shows CryoSPARC's processing output -- iteration progress, resolution estimates, and any error messages. For deeper debugging, use the `cryosparcm joblog` command (ask your admin if you have access).

### Finding CryoSPARC Jobs in Slurm

CryoSPARC names its Slurm jobs with a predictable pattern: `cryosparc_<project>_<job>`. You can find your CryoSPARC jobs using standard Slurm commands:

```bash
# See all your running and pending Slurm jobs
$ squeue -u $USER
  JOBID PARTITION     NAME     USER ST       TIME  NODES
 482901 gpu       cryospar user01  R    2:15:33      1
 482905 gpu       cryospar user01  R    0:45:12      1
 482910 gpu       cryospar user01 PD       0:00      1

# Filter to only CryoSPARC jobs
$ squeue -u $USER --name="cryosparc_*"

# Show full job name (default truncates it)
$ squeue -u $USER -o "%.10i %.12P %.40j %.8T %.10M %.4D %R"
  JOBID PARTITION  NAME                                     STATE       TIME NODE REASON
 482901 gpu        cryosparc_P12_J45                        RUNNING  2:15:33    1
 482905 gpu        cryosparc_P12_J47                        RUNNING  0:45:12    1
 482910 gpu        cryosparc_P15_J3                         PENDING     0:00    1 (Resources)
```

### Where to Find Cluster Logs

When CryoSPARC submits a job to Slurm, the standard output and error go to files in the job directory. The typical filenames follow the pattern:

```
<project_dir>/<job_dir>/<project_uid>_<job_uid>_slurm.out
<project_dir>/<job_dir>/<project_uid>_<job_uid>_slurm.err
```

For example:

```bash
# Find the Slurm output files for project P12, job J45
$ ls /data/cryosparc/CS-P12/J45/P12_J45_slurm.*
/data/cryosparc/CS-P12/J45/P12_J45_slurm.out
/data/cryosparc/CS-P12/J45/P12_J45_slurm.err
```

The exact filenames depend on your admin's cluster script template. Check the `.out` file for Slurm scheduling information and the `.err` file for any Slurm-level errors (resource exhaustion, node failures).

### Checking Completed Job Resources

After a CryoSPARC job finishes, you can check how many resources it actually used:

```bash
# Find the Slurm job ID from squeue history or the slurm.out file
$ sacct -j 482901 --format=JobID,JobName,Elapsed,MaxRSS,AllocCPUS,AllocGRES,State
  JobID      JobName    Elapsed     MaxRSS  AllocCPUS    AllocGRES      State
--------- ---------- ---------- ---------- ---------- ------------ ----------
  482901   cryospar+   02:15:33              8          gpu:1       COMPLETED
  482901.0     batch   02:15:33     42312M   8                      COMPLETED

# Quick summary (if seff is installed)
$ seff 482901
```

This is useful for understanding how much memory a particular job type actually consumed, which helps you or your admin right-size the cluster script template.

---

## Understanding Job Failures

When a CryoSPARC job fails, the cause may be in CryoSPARC itself, in Slurm, or in the interaction between them. Here is how to diagnose the source.

### Where to Look

| Log Source | How to Access | What It Tells You |
|-----------|---------------|-------------------|
| **CryoSPARC Event Log** | Click job card, press Spacebar | Processing errors, Python tracebacks |
| **CryoSPARC stdout log** | `cryosparcm joblog P12 J45` (ask admin) | Full standard output, more detail than Event Log |
| **Slurm output** | `<job_dir>/P12_J45_slurm.out` | Slurm scheduling info, node assignment |
| **Slurm error** | `<job_dir>/P12_J45_slurm.err` | Slurm-level errors, resource violations |
| **sacct** | `sacct -j <SLURM_JOBID>` | Exit codes, memory usage, time elapsed, state |

### CryoSPARC Errors vs. Slurm Errors

**CryoSPARC-side errors** appear in the Event Log and typically include a Python traceback ending with `Traceback (most recent call last)`:

- Invalid or missing inputs (empty required slots)
- Parameter misconfiguration
- Data file corruption
- Software bugs

**Slurm-side errors** appear in the `slurm.err` file or are visible via `sacct`:

- Resource allocation failures
- Out-of-memory kills
- Wall time exceeded
- Node failures

### Common Issues and Solutions

**GPU allocation failures:**
```
cuCtxCreate failed: invalid device ordinal
cuInit failed: unknown error
no CUDA-capable device is detected
```
- Slurm allocated the node but the GPU was not properly isolated
- This is a cluster configuration issue -- contact your admin
- The admin needs to verify that GRES and cgroup settings are correct

**GPU out of memory:**
```
cuMemAlloc failed: out of memory
cuArrayCreate failed: out of memory
cufftAllocFailed
```
- The particle box size exceeds what the GPU VRAM can handle
- A 12 GB GPU can typically handle box sizes up to ~700 pixels; 24 GB up to ~900
- Solutions: use Downsample Particles to reduce box size before refinement, or request a lane with larger-VRAM GPUs

**System memory (RAM) out of memory:**
```bash
# Check sacct -- look for OUT_OF_MEMORY state
$ sacct -j <SLURM_JOBID> --format=JobID,State,MaxRSS,ReqMem
```
- Slurm killed the job because it exceeded the requested RAM
- Ask your admin to increase the `ram_gb` multiplier in the cluster script, or configure a custom variable to adjust RAM per job

**SSD cache full:**
```
SSD cache : cache does not have enough space for download...
but there are no files that can be deleted.
```
- The local SSD on the compute node is full, often from other users' cached particles or non-CryoSPARC processes
- Contact your admin to clear stale cache entries or increase the `cache_reserve_mb` setting
- As a workaround, disable "Cache particle images on SSD" for the job (under Compute Settings), though this will slow down processing

**Wall time exceeded (TIMEOUT):**
```bash
$ sacct -j <SLURM_JOBID> --format=JobID,Elapsed,Timelimit,State
# State = TIMEOUT
```
- The Slurm partition has a time limit shorter than your job needed
- Large refinements or classifications with many particles can run for days
- Contact your admin about time limits on the GPU partition

**Job stuck in Queued status:**
- Check `squeue -u $USER` to see if the Slurm job is pending and why:
  ```bash
  $ squeue -u $USER -o "%.10i %.30j %.8T %R"
    JOBID NAME                           STATE    REASON
   482910 cryosparc_P15_J3               PENDING  (Resources)
  ```
- Common reasons: `(Resources)` means waiting for GPUs/nodes; `(Priority)` means other jobs are ahead in the queue
- In CryoSPARC v5.0+, the Queue panel shows lane utilization so you can choose a less-busy lane

### Diagnostic Checklist

1. **Job never started** -- check `squeue` for pending reason, verify the lane is configured correctly
2. **Job started and failed immediately** -- check CryoSPARC Event Log for input/parameter errors or GPU detection issues
3. **Job ran for a while then failed** -- check both the Event Log (for processing errors) and `sacct` (for OOM or TIMEOUT)
4. **Job completed but results look wrong** -- this is a CryoSPARC processing issue (parameters, data quality), not a Slurm issue

---

## Best Practices

### Project Directory Placement

CryoSPARC creates a self-contained project directory (e.g., `CS-P12/`) that holds all job outputs, metadata, and intermediate results. Proper placement matters:

- **Use a shared filesystem** (GPFS, Lustre, NFS) accessible from all cluster nodes. CryoSPARC requires the project directory to be visible at the same path on every node that runs jobs.
- **Use high-bandwidth storage** if possible. CryoSPARC preprocessing (motion correction, CTF estimation) needs sustained sequential reads of ~1000 MB/s to saturate a 4-GPU node.
- **Watch your disk quota.** A single cryo-EM project can easily consume 1-5 TB. Motion-corrected micrographs, extracted particles, and refinement iterations all add up.
- **Clean up regularly.** Use CryoSPARC's built-in Cleanup Data tool (v4.3+) to clear intermediate results, prune failed jobs, and compact Live sessions. Clearing intermediate results from iterative jobs (like refinement) can recover significant space while keeping final outputs.

> **ParallelCluster Note:** On AWS ParallelCluster, project directories should live on FSx for Lustre or EFS, not on the head node's local storage. FSx for Lustre provides the throughput needed for preprocessing and can outperform local NVMe for multi-GPU 2D classification workloads.

### SSD Caching

SSD caching is one of the most important performance features in CryoSPARC. Classification, refinement, and reconstruction jobs perform random-access reads across particle images, which is extremely slow on spinning disks or network storage. Local SSDs on compute nodes eliminate this bottleneck.

**How it works:**
- When SSD caching is enabled, CryoSPARC copies particle images from the project directory to a local SSD (`/scratch`, `/tmp/ssd`, or similar) on the compute node before processing
- Subsequent jobs reuse the cached particles if they are still on the SSD
- CryoSPARC automatically manages the cache, removing files not accessed in 30 days (configurable)

**When to enable it:** Always keep "Cache particle images on SSD" enabled (under Compute Settings) for:
- 2D Classification
- Ab-Initio Reconstruction
- Homogeneous / Non-Uniform / Heterogeneous Refinement
- 3D Variability Analysis

**When it does not apply:** Preprocessing jobs (motion correction, CTF estimation, particle picking) read data sequentially and do not benefit from SSD caching. These jobs are I/O-bound on sequential read throughput, not random access.

**SSD sizing:** A 1M-particle dataset with box size 256 is ~263 GB. A 2M-particle dataset with box size 432 is ~1.5 TB. Your admin should configure SSDs of 1-2 TB per compute node.

### Resource Estimation for Common Job Types

Use this table as a rough guide when planning your processing campaigns. Actual runtimes depend on dataset size, box size, and GPU hardware.

| Job Type | GPUs | Typical Runtime | Memory Profile | Notes |
|----------|------|----------------|----------------|-------|
| Patch Motion Correction | 2-8 | Minutes per 1000 movies | Moderate (~15 GB/GPU) | Scales well across GPUs; I/O-bound |
| Patch CTF Estimation | 2-8 | Minutes per 1000 micrographs | Low | Scales well across GPUs |
| 2D Classification | 1-2 | 10-60 min (100K-1M particles) | Moderate | Benefits from SSD cache |
| Ab-Initio Reconstruction | 1-3 | 30-120 min | Moderate | 1 GPU per class |
| Homogeneous Refinement | 1 | 30-120 min | Moderate to high | Single GPU; VRAM limits box size |
| Non-Uniform Refinement | 1 | 60-180 min | High | Single GPU; larger memory footprint |
| Heterogeneous Refinement | 1-5 | 60-240 min | Moderate per class | 1 GPU per class |
| Local Refinement | 1 | 30-120 min | Moderate | Single GPU |
| 3D Variability Analysis | 1 | 60-240 min | Moderate | Single GPU |

### Avoiding Resource Waste on GPU Nodes

GPU nodes are the most expensive resources on any cluster. A few practices help maximize utilization:

- **Do not request GPUs for CPU-only jobs.** Import, Manual Curation, and some utility jobs do not need GPUs. Run these on a CPU lane or the workstation lane if your admin has one configured.
- **Match GPU count to job type.** Single-GPU jobs (refinement, ab-initio per class) do not benefit from extra GPUs. Check the table above.
- **Queue preprocessing in batches.** If you have thousands of movies, queue a multi-GPU preprocessing job rather than many single-GPU jobs. This reduces Slurm scheduling overhead.
- **Clear and re-queue rather than hoarding slots.** If you realize a job has wrong parameters, kill it promptly rather than letting it run to completion and wasting GPU hours.
- **Use priority queuing thoughtfully.** CryoSPARC supports job priorities. Do not set all your jobs to the highest priority -- it defeats the purpose of fair scheduling across users.

### Working with Your Admin

As a CryoSPARC user on a Slurm cluster, some things are outside your direct control. Know when to ask your admin for help:

- **Lane configuration:** If you need access to a specific GPU type or partition, your admin manages lane assignments
- **Resource limits:** If jobs consistently fail with OOM or TIMEOUT, the cluster script template may need tuning
- **SSD cache issues:** If SSD caching is slow or showing errors, the admin needs to check SSD health and configuration
- **Software updates:** CryoSPARC updates can include performance improvements and bug fixes; coordinate with your admin on upgrade timing

CryoSPARC Admin Setup -- how your admin configures CryoSPARC for Slurm
Monitoring Jobs -- general Slurm job monitoring
GPU Jobs -- Slurm GPU job concepts

CryoSPARC User

Related Modules¶

References¶