CryoSPARC Admin

<!-- audience: L4 --> <!-- deployment: all --> <!-- sge-source: none --> # CryoSPARC & Slurm -- Admin Setup ## Overview CryoSPARC is a high-performance computing platform for single-particle cryo-electron microscopy (cryo-EM) structure determination. It provides GPU-accelerated algorithms for motion correction, CTF estimation, particle picking, 2D classification, 3D reconstruction, and refinement -- all accessible through a browser-based interface. CryoSPARC uses a **master-worker architecture** with its own internal job scheduler. It does not replace Slurm; instead, it submits jobs *through* Slurm using templated submission scripts. Understanding how these two scheduling layers interact is the key to a successful deployment. ### Architecture on a Slurm Cluster | Component | Role | Typical Location | |---|---|---| | **CryoSPARC Master** | Web app, MongoDB database, internal scheduler | Lightweight head node (no GPU required) | | **CryoSPARC Worker** | GPU computation engine (`cryosparcw`) | Installed on shared storage, runs on GPU compute nodes | | **Slurm** | Resource allocation, GPU isolation, job queuing | Cluster-wide (slurmctld on head node, slurmd on computes) | | **Shared Filesystem** | Project data, raw movies, intermediate results | NFS, GPFS, BeeGFS, or Lustre mounted on all nodes | | **Local SSD Cache** | Fast random-access particle I/O during reconstruction | NVMe on each GPU node (e.g., `/scratch`) | CryoSPARC's internal scheduler decides *when* a job is ready to run. Slurm decides *where* (which node, which GPUs) and enforces resource isolation. The bridge between them is a **cluster lane** -- a pair of configuration files (`cluster_info.json` and `cluster_script.sh`) that define how CryoSPARC constructs and submits `sbatch` commands. ### Lanes A **lane** in CryoSPARC is a named compute target. There are two types: - **Worker lane**: Direct SSH dispatch to a managed worker node (no Slurm involvement) - **Cluster lane**: Jobs submitted via Slurm (or PBS, SGE, etc.) using templated scripts For shared HPC clusters, cluster lanes are the standard approach. You can define multiple lanes targeting different Slurm partitions, GPU types, or resource profiles. --- ## Prerequisites ### 1. NVIDIA GPU Driver CryoSPARC v5.0+ requires **NVIDIA driver version 570.26 or newer**. The software ships with CUDA 12.8 internally, so a separate system CUDA installation is not required. Supported GPU compute capabilities range from 5.0 (Maxwell) through 12.0 (Blackwell). For older CryoSPARC versions: | CryoSPARC Version | Minimum Driver | Bundled CUDA | |---|---|---| | v5.0+ | 570.26 | 12.8 | | v4.4 -- v4.7 | 520.61.05 | 11.8 | | <= v4.3 | Varies | Requires system CUDA 11.x | Verify the driver on each GPU node: ```bash nvidia-smi --query-gpu=driver_version,name,memory.total --format=csv,noheader # 570.86.16, NVIDIA A100-SXM4-80GB, 81920 MiB ``` ### 2. Common Unix User Account The same non-privileged Linux account must exist on the CryoSPARC master node and all worker/compute nodes, with the **same numeric UID** everywhere. This account (conventionally `cryosparc`) owns all CryoSPARC processes and files. Do **not** install or run CryoSPARC as root. ```bash # Verify consistent UID across nodes pdsh -w gpu[01-08] "id cryosparc" # uid=2001(cryosparc) gid=2001(cryosparc) ... ``` ### 3. Password-less SSH Access The CryoSPARC master node must have password-less SSH access (via SSH key-pair) to all worker nodes under the `cryosparc` account. This is used for managed (non-cluster) worker dispatch. For cluster lanes submitted via Slurm, SSH between the master and compute nodes is not strictly required -- Slurm handles job dispatch. However, SSH is needed for `cryosparcm test workers` and other management operations. ```bash # As the cryosparc user on the master node ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519 -N "" ssh-copy-id cryosparc@gpu01 ``` ### 4. Shared Filesystem All nodes (master + compute) must have access to the same shared filesystem where CryoSPARC project directories reside. Raw movie files, intermediate results, and final maps are read and written here. | Path | Purpose | Performance Need | |---|---|---| | `/shared/cryosparc_master/` | Master installation + database | Moderate IOPS | | `/shared/cryosparc_worker/` | Worker installation (read-only at runtime) | Low | | `/shared/projects/` | Project data (movies, particles, maps) | High throughput (10+ Gbps recommended) | ### 5. Network Ports All worker/compute nodes must have TCP access to **10 consecutive ports** on the master node (default: **39000--39009**). These ports handle HTTP RPC communication between the master and workers. ```bash # Verify connectivity from a compute node for port in $(seq 39000 39009); do nc -zv cryosparc-master $port 2>&1 | grep -q succeeded && echo "Port $port: OK" done ``` ### 6. Local SSD Cache CryoSPARC reconstruction jobs benefit significantly from fast local storage for particle caching. Each GPU compute node should have an NVMe SSD with a dedicated cache directory: - **Minimum**: 1 TB SSD per node - **Recommended**: 2 TB PCIe NVMe SSD per node - The path (e.g., `/scratch/cryosparc_cache`) must be identical on all compute nodes - Writable by the `cryosparc` user --- ## Slurm-Specific Configuration ### GRES for GPUs CryoSPARC relies on Slurm's Generic Resource (GRES) system to request and isolate GPUs. Correct GRES configuration is **essential** for CryoSPARC to function on a shared cluster. #### slurm.conf Enable the GRES plugin and declare GPU resources on each node: ```ini # slurm.conf GresTypes=gpu # Node definitions with GPU GRES NodeName=gpu[01-04] Gres=gpu:a100:4 CPUs=64 RealMemory=512000 State=UNKNOWN NodeName=gpu[05-08] Gres=gpu:a6000:2 CPUs=32 RealMemory=256000 State=UNKNOWN ``` #### gres.conf Map each GPU to its device file. Using `AutoDetect=nvml` is recommended for NVIDIA GPUs: ```ini # gres.conf -- Option 1: AutoDetect (recommended) AutoDetect=nvml # gres.conf -- Option 2: Explicit (for heterogeneous or problematic nodes) NodeName=gpu[01-04] Name=gpu Type=a100 File=/dev/nvidia[0-3] NodeName=gpu[05-08] Name=gpu Type=a6000 File=/dev/nvidia[0-1] ``` Verify detection on a compute node: ```bash slurmd -C # NodeName=gpu01 CPUs=64 ... Gres=gpu:a100_sxm4_80gb:4 ... ``` ### Cgroup GPU Isolation Cgroup enforcement is **critical** for CryoSPARC. Without it, multiple CryoSPARC jobs on the same node will compete for the same GPU device indices and corrupt each other's results. CryoSPARC jobs always request GPUs starting at device index 0. Slurm's cgroup device isolation remaps `CUDA_VISIBLE_DEVICES` so that each job sees only its allocated GPUs, renumbered from 0. ```ini # cgroup.conf ConstrainCores=yes ConstrainDevices=yes ConstrainRAMSpace=yes ``` ```ini # slurm.conf ProctrackType=proctrack/cgroup TaskPlugin=task/cgroup,task/affinity ``` ### Partition Design for GPU Workloads Design partitions to match CryoSPARC workload patterns: ```ini # slurm.conf partition examples PartitionName=gpu Nodes=gpu[01-08] MaxTime=7-00:00:00 Default=NO PartitionName=gpu-a100 Nodes=gpu[01-04] MaxTime=7-00:00:00 Default=NO PartitionName=gpu-a6000 Nodes=gpu[05-08] MaxTime=3-00:00:00 Default=NO ``` Multiple partitions let you create separate CryoSPARC lanes for different GPU types, which is useful since cryo-EM job types have different GPU memory and bandwidth requirements. ### Memory Configuration CryoSPARC recommends **32 GB RAM minimum per GPU**, with 64 GB per GPU preferred. Ensure `RealMemory` in `slurm.conf` accurately reflects available memory and that `DefMemPerCPU` or `DefMemPerNode` are set sensibly: ```ini # slurm.conf DefMemPerCPU=8000 # Ensure CryoSPARC jobs can request enough memory MaxMemPerNode=UNLIMITED ``` --- ## Cluster Lane Setup ### How It Works When CryoSPARC queues a job to a cluster lane, it: 1. Renders `cluster_script.sh` using Jinja2, substituting job-specific variables 2. Writes the rendered script to the job directory 3. Executes the `qsub_cmd_tpl` from `cluster_info.json` (i.e., `sbatch`) 4. Monitors the job using `qstat_cmd_tpl` (i.e., `squeue`) 5. Can cancel using `qdel_cmd_tpl` (i.e., `scancel`) ### cluster_info.json This file defines the Slurm commands CryoSPARC uses, as Jinja2 templates: ```json { "name": "slurm-gpu", "worker_bin_path": "/shared/cryosparc_worker/bin/cryosparcw", "cache_path": "/scratch/cryosparc_cache", "cache_quota_mb": 1500000, "cache_reserve_mb": 10000, "send_cmd_tpl": "{{ command }}", "qsub_cmd_tpl": "/usr/bin/sbatch {{ script_path_abs }}", "qstat_cmd_tpl": "/usr/bin/squeue -j {{ cluster_job_id }}", "qdel_cmd_tpl": "/usr/bin/scancel {{ cluster_job_id }}", "qinfo_cmd_tpl": "/usr/bin/sinfo" } ``` | Field | Description | |---|---| | `name` | Unique lane name (appears in the CryoSPARC UI when queuing jobs) | | `worker_bin_path` | Absolute path to `cryosparcw` on compute nodes | | `cache_path` | Local SSD path on compute nodes for particle caching | | `cache_quota_mb` | Maximum SSD space CryoSPARC may use (in MB) | | `cache_reserve_mb` | Minimum free space to leave on the SSD (default: 10 GB) | | `send_cmd_tpl` | How to send cluster commands; `{{ command }}` if master can run sbatch directly | | `qsub_cmd_tpl` | Job submission command (`sbatch`) | | `qstat_cmd_tpl` | Job status command (`squeue`) | | `qdel_cmd_tpl` | Job cancellation command (`scancel`) | | `qinfo_cmd_tpl` | Cluster info command (`sinfo`) | If the CryoSPARC master cannot directly execute Slurm commands (e.g., it is not a Slurm submit host), use `send_cmd_tpl` to proxy through SSH: ```json "send_cmd_tpl": "ssh login01 {{ command }}" ``` ### cluster_script.sh This is the Jinja2 template for the Slurm submission script. CryoSPARC renders it for each job before calling `sbatch`: ```bash #!/usr/bin/env bash #SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }} #SBATCH --partition=gpu #SBATCH --cpus-per-task={{ num_cpu }} #SBATCH --gres=gpu:{{ num_gpu }} #SBATCH --mem={{ ram_gb|int }}G #SBATCH --comment="created by {{ cryosparc_username }}" #SBATCH --output={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.out #SBATCH --error={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.err {{ run_cmd }} ``` ### Template Variables Reference These are reserved variables that CryoSPARC injects into the template at render time: | Variable | Description | |---|---| | `{{ run_cmd }}` | Complete command string to execute the CryoSPARC job | | `{{ num_cpu }}` | Number of CPUs needed | | `{{ num_gpu }}` | Number of GPUs needed | | `{{ ram_gb }}` | RAM needed in GB | | `{{ job_dir_abs }}` | Absolute path to the job directory | | `{{ project_dir_abs }}` | Absolute path to the project directory | | `{{ job_log_path_abs }}` | Absolute path to the job log file | | `{{ worker_bin_path }}` | Absolute path to `cryosparcw` | | `{{ run_args }}` | Arguments passed to `cryosparcw run` | | `{{ project_uid }}` | Project UID (e.g., `P3`) | | `{{ job_uid }}` | Job UID (e.g., `J42`) | | `{{ job_creator }}` | Name of the user who created the job (may contain spaces) | | `{{ cryosparc_username }}` | CryoSPARC username (usually an email) | | `{{ job_type }}` | CryoSPARC job type identifier | | `{{ script_path_abs }}` | Absolute path to the rendered submission script | | `{{ cluster_job_id }}` | Slurm job ID (available after submission, used in `qstat`/`qdel` templates) | ### Registering, Updating, and Removing Lanes ```bash # Generate example Slurm templates in the current directory cryosparcm cluster example slurm # Edit cluster_info.json and cluster_script.sh, then register cryosparcm cluster connect # Dump an existing lane's config to the current directory cryosparcm cluster dump slurm-gpu # Update: edit files in-place, then re-run connect (same name = update) cryosparcm cluster connect # Remove a lane cryosparcm cluster remove slurm-gpu ``` ### Multiple Lanes for Different GPU Types Create separate directories, each containing a `cluster_info.json` and `cluster_script.sh` with different `name` values and partition targets: ```bash mkdir -p ~/cryosparc-lanes/{a100-lane,a6000-lane} # a100-lane/cluster_info.json -- name: "slurm-a100" # a100-lane/cluster_script.sh -- #SBATCH --partition=gpu-a100 --gres=gpu:a100:{{ num_gpu }} # a6000-lane/cluster_info.json -- name: "slurm-a6000" # a6000-lane/cluster_script.sh -- #SBATCH --partition=gpu-a6000 --gres=gpu:a6000:{{ num_gpu }} cd ~/cryosparc-lanes/a100-lane && cryosparcm cluster connect cd ~/cryosparc-lanes/a6000-lane && cryosparcm cluster connect ``` Users then see both lanes in the CryoSPARC job queuing interface and can choose the appropriate GPU type for each job. --- ## Custom Variables CryoSPARC v4.1+ supports user-defined custom variables that are injected into `cluster_script.sh` before submission. This is useful for adjusting resource requests on the fly without editing the lane configuration. ### Precedence Hierarchy Custom variables can be set at three levels, with later levels overriding earlier ones: 1. **Instance level** -- Applies to all cluster lanes (set in Admin Panel > Cluster Configuration) 2. **Target level** -- Applies to a specific lane (set per-target in Admin Panel) 3. **Job level** -- Set by the user when queuing each individual job ### Example: ram_multiplier CryoSPARC's `ram_gb` estimate is sometimes too low for large particle boxes. A `ram_multiplier` custom variable lets users request more memory without modifying the lane: ```bash #!/usr/bin/env bash #SBATCH --chdir={{ job_dir_abs }} #SBATCH --export=NONE #SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }} #SBATCH --partition=gpu #SBATCH --cpus-per-task={{ num_cpu }} #SBATCH --gres=gpu:{{ num_gpu }} #SBATCH --mem={{ (ram_multiplier | default(1) | float * ram_gb) | int }}G #SBATCH --output={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.out #SBATCH --error={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.err {{ run_cmd }} ``` Note that custom variable values are injected as **strings**. Use Jinja2 filters like `|float` or `|int` when arithmetic is required. ### Example: Slurm Account or QOS ```bash #SBATCH --account={{ slurm_account | default("cryoem") }} #SBATCH --qos={{ slurm_qos | default("normal") }} #SBATCH --time={{ time_limit | default("7-00:00:00") }} ``` Set defaults at the instance level, and let users override per-job when submitting to different allocations. --- ## GPU Resource Management This is the most critical concept for CryoSPARC on Slurm. Misunderstanding how GPU allocation works is the single most common source of deployment failures. ### How CryoSPARC Sees GPUs CryoSPARC does **not** know which specific GPU devices Slurm allocates. When a job requests 2 GPUs, CryoSPARC simply uses devices starting at index 0: `[0, 1]`. It is entirely the responsibility of Slurm to: 1. Allocate the requested number of GPUs 2. Set `CUDA_VISIBLE_DEVICES` so that only the allocated GPUs are visible 3. Remap device indices so the job sees them as `[0, 1, ...]` ### Why Cgroups Are Essential Without cgroup device isolation, Slurm sets `CUDA_VISIBLE_DEVICES` as an environment variable but does not prevent processes from accessing other GPU devices. With cgroup enforcement (`ConstrainDevices=yes` in `cgroup.conf`), Slurm restricts the job's device namespace at the kernel level, making only the allocated GPUs visible. This is the only reliable method for GPU isolation. The remapping behavior changes with cgroups: | Scenario | Allocated GPUs | CUDA_VISIBLE_DEVICES | |---|---|---| | Without cgroups | /dev/nvidia2, /dev/nvidia3 | `2,3` | | With cgroups | /dev/nvidia2, /dev/nvidia3 | `0,1` (renumbered) | CryoSPARC expects the second behavior (starting from 0), which is why **GRES + cgroup integration is essential**. ### Configuration Checklist ``` [ ] GresTypes=gpu in slurm.conf [ ] NodeName entries include Gres=gpu:<type>:<count> [ ] gres.conf has File= entries or AutoDetect=nvml [ ] cgroup.conf has ConstrainDevices=yes [ ] ProctrackType=proctrack/cgroup in slurm.conf [ ] TaskPlugin includes task/cgroup [ ] cluster_script.sh includes --gres=gpu:{{ num_gpu }} ``` ### Verifying GPU Isolation Test that Slurm correctly isolates GPUs before connecting CryoSPARC: ```bash # Submit two single-GPU jobs simultaneously srun --partition=gpu --gres=gpu:1 --ntasks=1 \ bash -c 'echo "Job $SLURM_JOB_ID: CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES"' & srun --partition=gpu --gres=gpu:1 --ntasks=1 \ bash -c 'echo "Job $SLURM_JOB_ID: CUDA_VISIBLE_DEVICES=$CUDA_VISIBLE_DEVICES"' & wait # Both jobs should report CUDA_VISIBLE_DEVICES=0 (remapped via cgroups) # They should be on different physical GPUs ``` --- ## Testing and Validation ### cryosparcm test install Verifies core components: HTTP connections, licensing, database, and worker connectivity. ```bash cryosparcm test install ``` This checks that the master is running, all services are reachable, and the license is valid. Run this first before testing workers or cluster lanes. ### cryosparcm test workers Tests that worker nodes can launch jobs, cache particles to the SSD, and utilize GPUs correctly: ```bash cryosparcm test workers P1 # Requires an existing project (P1) to run test jobs against ``` ### Extensive Validation Job The most thorough validation is to create an **Extensive Validation** job in CryoSPARC and queue it to the cluster lane. This runs a complete end-to-end processing workflow (import, motion correction, CTF, picking, 2D classification, 3D reconstruction) and verifies: - Slurm submission and monitoring work - GPU access and CUDA operations succeed - SSD caching functions correctly - Shared filesystem I/O is adequate - Master-worker communication over the network works To run: Create an Extensive Validation job in the CryoSPARC UI and select your cluster lane when queuing. ### Inspecting Lane Configurations After registering lanes, verify what CryoSPARC has stored and what users will see: ```bash # List all registered cluster lanes with their names cryosparcm cli "get_scheduler_lanes()" # Returns a list of lane objects -- the 'name' field matches # what users see in the Queue panel dropdown # Dump a specific lane's config files to the current directory cryosparcm cluster dump slurm-a100 ls # cluster_info.json cluster_script.sh # Inspect the Slurm arguments embedded in the lane cat cluster_script.sh # Shows #SBATCH directives: --partition, --gres, --mem, etc. # This is the template CryoSPARC renders for every job on this lane ``` ### Viewing Rendered Submission Scripts After a job has been queued, CryoSPARC writes the rendered (variable-substituted) submission script to the job directory. This is essential for debugging -- it shows the exact `sbatch` command that Slurm received: ```bash # Find the rendered script for a specific job ls /path/to/project/J42/ # cluster_job.sh -- the rendered sbatch script # P3_J42_slurm.out -- Slurm stdout # P3_J42_slurm.err -- Slurm stderr cat /path/to/project/J42/cluster_job.sh # #!/usr/bin/env bash # #SBATCH --job-name cryosparc_P3_J42 # #SBATCH --partition=gpu-a100 # #SBATCH --cpus-per-task=8 # #SBATCH --gres=gpu:a100:2 # #SBATCH --mem=64G # ... # /shared/cryosparc_worker/bin/cryosparcw run ... ``` Compare the rendered script against the lane template to verify variables were substituted correctly. Common issues to look for: - `--partition` pointing at the wrong Slurm partition - `--gres=gpu:{{ num_gpu }}` not substituted (indicates a Jinja2 template error) - Missing `--mem` or incorrect memory calculation (check `ram_multiplier` if used) ### Quick Validation Checklist ```bash # 1. Verify CryoSPARC services are running cryosparcm status # 2. Check the cluster lane is registered cryosparcm cli "get_scheduler_lanes()" # 3. Dump and review the lane's Slurm arguments cryosparcm cluster dump slurm-a100 cat cluster_script.sh | grep SBATCH # 4. Verify Slurm can see the target partition and GPUs sinfo -p gpu-a100 -o "%P %a %l %D %T %G" # 5. Dry-run a submission to check Slurm constraints sbatch --test-only --partition=gpu-a100 --gres=gpu:a100:1 --wrap="hostname" # 6. Check a submitted CryoSPARC job appears in Slurm squeue -u cryosparc --format="%.8i %.30j %.8T %.10M %.6D %R" # 7. After a job completes, verify GPU usage and resources sacct -j <JOBID> --format=JobID,JobName,AllocTRES%40,Elapsed,MaxRSS,ExitCode # 8. Inspect the rendered submission script cat /path/to/project/J42/cluster_job.sh ``` --- ## CryoSPARC Live on Slurm CryoSPARC Live enables real-time cryo-EM data processing during microscope data collection sessions. It has three task types, each of which can be assigned to a different lane: | Task Type | Description | GPU Requirement | |---|---|---| | **Preprocessing** | Motion correction, CTF estimation, picking, extraction | 1+ GPUs (scales linearly) | | **Reconstruction** | Streaming 2D classification and 3D refinement | 1 GPU per task (runs indefinitely) | | **Auxiliary** | Template generation, Ab-Initio models | 1 GPU per job | **Minimum recommendation**: 4 GPUs total for a seamless live session. ### Slurm Considerations for Live CryoSPARC Live workers run **indefinitely** until the session is paused. This creates challenges with Slurm's time limits and QOS policies: - **Time limits**: Set `MaxTime` high enough on the GPU partition (e.g., `7-00:00:00` or `UNLIMITED`) or use a dedicated Live partition with no time limit - **QOS policies**: If your cluster enforces job count or runtime limits, CryoSPARC Live may hit those limits. Consider a dedicated QOS for CryoSPARC Live jobs - **Cluster scheduling latency**: Jobs must go through the Slurm queue, adding startup latency. For the lowest-latency Live experience, consider using a managed worker node (bypasses Slurm) for preprocessing ### Dedicated Node Strategy For facilities where CryoSPARC Live is used frequently, the recommended approach is to configure dedicated GPU nodes as managed workers (direct SSH, no Slurm) for Live sessions, while keeping cluster lanes for batch processing. This avoids Slurm scheduling delays and QOS conflicts. ```bash # Connect a dedicated Live worker (bypasses Slurm) cd /shared/cryosparc_worker bin/cryosparcw connect \ --worker gpu-live01 \ --master cryosparc-master \ --port 39000 \ --ssdpath /scratch/cryosparc_cache \ --lane live ``` --- ## Production cluster_script.sh Examples ### Basic Slurm Template ```bash #!/usr/bin/env bash #SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }} #SBATCH --partition=gpu #SBATCH --cpus-per-task={{ num_cpu }} #SBATCH --gres=gpu:{{ num_gpu }} #SBATCH --mem={{ ram_gb|int }}G #SBATCH --output={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.out #SBATCH --error={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.err {{ run_cmd }} ``` ### Production-Hardened Template ```bash #!/usr/bin/env bash #SBATCH --chdir={{ job_dir_abs }} #SBATCH --export=NONE #SBATCH --job-name cs_{{ project_uid }}_{{ job_uid }} #SBATCH --partition={{ slurm_partition | default("gpu") }} #SBATCH --account={{ slurm_account | default("cryoem") }} #SBATCH --cpus-per-task={{ num_cpu }} #SBATCH --gres=gpu:{{ num_gpu }} #SBATCH --mem={{ (ram_multiplier | default(1.2) | float * ram_gb) | int }}G #SBATCH --time={{ time_limit | default("3-00:00:00") }} #SBATCH --output={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.out #SBATCH --error={{ job_dir_abs }}/{{ project_uid }}_{{ job_uid }}_slurm.err #SBATCH --comment="cryosparc {{ job_type }} by {{ cryosparc_username }}" # Ensure consistent GPU ordering export CUDA_DEVICE_ORDER=PCI_BUS_ID {{ run_cmd }} ``` Key enhancements in the production template: - `--export=NONE` prevents inheriting the submitter's environment (avoids PATH conflicts) - `--chdir` sets the working directory to the job directory - `ram_multiplier` defaults to 1.2x (CryoSPARC estimates are often slightly low) - `time_limit` is configurable per-job via custom variables - `CUDA_DEVICE_ORDER=PCI_BUS_ID` ensures CUDA device numbering matches Slurm's PCI-bus-based GRES ordering --- ## Deployment Callouts > **ParallelCluster Note:** On AWS ParallelCluster, CryoSPARC works well with GPU instances across multiple queues. Recommended instance types: **g4dn.12xlarge** (cost-effective preprocessing), **p4d.24xlarge** (high-throughput reconstruction with 8x A100), and **p3.16xlarge** (good all-around). Use **Amazon FSx for Lustre** as the shared filesystem -- it provides the throughput needed for cryo-EM I/O and can be linked to S3 for raw data staging. ParallelCluster's multiple queue feature (v2.9+) maps naturally to CryoSPARC lanes: define `gpu-large`, `gpu-med`, and `gpu-small` queues with different instance types and create a CryoSPARC lane for each. Set `MinCount=0` and `MaxCount` for compute nodes to enable scale-to-zero when idle. The `worker_bin_path` in `cluster_info.json` must point to a path on shared storage accessible from dynamically launched instances. Note: NVIDIA drivers are pre-installed on ParallelCluster GPU AMIs; verify the version meets CryoSPARC requirements (570.26+ for v5.0+). > **PCS Note:** AWS PCS manages Slurm configuration for you. Install CryoSPARC on shared storage (EFS or FSx) accessible from all compute node groups. Create compute node groups with GPU instances and configure CryoSPARC cluster lanes to target the appropriate PCS queue names. Ensure security groups allow TCP access on ports 39000--39009 between the master and compute nodes. > **On-Prem Note:** On-prem clusters offer the most configuration flexibility. Mount the CryoSPARC installation via NFS/GPFS/BeeGFS, use local NVMe for the SSD cache path, and ensure cgroup GPU isolation is properly configured. For multi-GPU heterogeneous nodes, use explicit `gres.conf` entries with `Type=` to differentiate GPU models, and create separate CryoSPARC lanes for each type. Enable NVIDIA driver persistence mode (`nvidia-smi --pm 1`) on all GPU nodes to reduce driver load times. --- ## Troubleshooting Quick Reference | Symptom | Likely Cause | Fix | |---|---|---| | Jobs stuck in "Queued" | Slurm cannot run `sbatch` from the master node | Check `send_cmd_tpl`; verify Slurm client is installed on the master or use SSH proxy | | Jobs start but immediately fail | Wrong `worker_bin_path` | Verify path is accessible from compute nodes; check shared filesystem mount | | GPU errors (`cudaErrorInsufficientDriver`) | Driver too old for CryoSPARC version | Upgrade NVIDIA driver to >= 570.26 for v5.0+ | | Two jobs fighting over same GPU | Cgroup device isolation not enabled | Set `ConstrainDevices=yes` in `cgroup.conf` | | SSD cache errors | `cache_path` not writable or full | Verify path exists on all compute nodes; check `cache_quota_mb` and `cache_reserve_mb` | | Job log says `CUDA_VISIBLE_DEVICES` empty | GRES not configured or `--gres` missing from script | Verify `gres.conf`, GRES in `slurm.conf`, and `--gres=gpu:{{ num_gpu }}` in template | | Master cannot monitor job status | `squeue` path wrong or not accessible | Verify `qstat_cmd_tpl` uses the correct absolute path to `squeue` |

Related Modules

• CryoSPARC User Guide -- end-user perspective
• Resource Management -- GRES and cgroup configuration
• GPU Jobs -- Slurm GPU scheduling concepts
• On-Premise Deployment -- cluster infrastructure for cryo-EM

References

• CryoSPARC: Architecture and System Requirements
• CryoSPARC: Installation Prerequisites
• CryoSPARC: Downloading and Installing
• CryoSPARC: Cluster Integration Script Examples
• CryoSPARC: Custom Variables for Cluster Submission Scripts
• CryoSPARC: Installation Testing with cryosparcm test
• CryoSPARC: Extensive Validation Tutorial
• CryoSPARC: CryoSPARC Live Prerequisites
• CryoSPARC: Deploying on AWS with ParallelCluster
• CryoSPARC: Performance Benchmarks on AWS
• SchedMD: Generic Resource (GRES) Scheduling
• SchedMD: gres.conf Configuration
• SchedMD: Cgroup Support