What is HPC Scheduling?

<!-- audience: L1, L5 --> <!-- deployment: all --> <!-- sge-source: SGE User "DRM Intro" + Admin "Overview" --> # What is HPC Scheduling? --- ## Why Do We Need a Scheduler? Modern research computing relies on shared infrastructure -- clusters of servers with hundreds or thousands of CPU cores, terabytes of memory, and specialized hardware like GPUs. Multiple researchers, groups, and projects all need access to these resources. Without a traffic cop, shared computing looks like this: - Users SSH directly into individual servers - The most aggressive users grab the most resources - No way to enforce priorities or fairness - Servers get overloaded while others sit idle - "Consensus" management breaks down as the user base grows - Business and scientific priorities are not reflected in how resources are used **This is the "old way" -- and it doesn't scale.** --- ## A Better Way: Distributed Resource Management A **workload manager** (also called a job scheduler or DRM -- Distributed Resource Manager) sits between users and compute resources. Instead of users picking machines directly, they submit work requests to the scheduler, which: 1. **Accepts work requests (jobs)** from users 2. **Queues jobs** in a pending area 3. **Matches jobs to the best available resources** based on requirements 4. **Manages jobs** while they run 5. **Returns results** and logs accounting data when jobs finish --- ## From the User's Perspective A scheduler **simplifies your life**: - You interact with one system, not individual machines - You don't need to know or care which specific server runs your job - You describe what your job needs (CPUs, memory, GPUs, time) and the scheduler finds the right place to run it - Your work is treated fairly relative to other users --- ## From the Organization's Perspective A scheduler lets management **align computing with business priorities**: - Scientific or business priorities can dynamically influence resource allocation - Prevents inequity caused by aggressive users monopolizing systems - Maximizes infrastructure utilization through intelligent load balancing - Provides detailed accounting of who used what, when, and how much - Enables chargeback/showback for cost allocation --- ## "The Contract" This is the most important concept in scheduler training: > **Your responsibility:** Describe what resources your job needs to run successfully. > > **The scheduler's responsibility:** Match your job to available resources and get results back to you. The better you describe your job's requirements, the better the scheduler can place it. In most cases, the defaults work fine. But when you need specific resources -- a GPU, extra memory, exclusive access to a node -- you tell the scheduler, and it handles the rest. --- ## Types of HPC Workloads ### Embarrassingly Parallel (Batch/Serial Computing) The most common pattern in life science HPC: - Large numbers of nearly identical jobs - Each job is independent -- only the input data changes - Examples: BLAST searches across sequences, variant calling per sample, docking thousands of compounds - The scheduler's job: distribute these tasks across available nodes efficiently --- ### Tightly Coupled (Parallel Computing) Traditional HPC parallelism: - A single program uses multiple processors that must communicate - Uses MPI (Message Passing Interface) or similar frameworks - Examples: molecular dynamics (GROMACS, NAMD), weather modeling, computational fluid dynamics - The scheduler's job: allocate the right number of processors, ideally on closely-connected nodes ### Mixed Workloads Most real clusters support both types. The scheduler must balance: - Large parallel jobs that need many nodes reserved together - Many small serial jobs that can fill gaps - GPU workloads (cryo-EM, deep learning, molecular dynamics) - Interactive sessions for development and visualization --- ## Life Science Examples | Workload | Type | Typical Resources | |----------|------|------------------| | BLAST sequence search | Embarrassingly parallel | 1-4 CPUs per query, many queries | | Genome assembly | Mixed | High memory, multiple CPUs | | Cryo-EM reconstruction (CryoSPARC/RELION) | GPU-accelerated | Multiple GPUs, high memory | | Molecular dynamics (Desmond, GROMACS) | Tightly coupled / GPU | Multiple nodes or GPUs | | Docking (Glide) | Embarrassingly parallel | Many single-CPU jobs | | RNA-seq pipeline | Workflow of serial steps | Varies by step, job dependencies | | AlphaFold protein prediction | GPU-accelerated | 1+ GPUs, high memory | --- ## What Slurm Does for You Slurm is the scheduler you'll be learning in this training. At a high level, it: - Allows multiple users, groups, and projects to work together on shared infrastructure - Lets scientific and research priorities be reflected in how the system is used - Treats you fairly - Stays out of your way The key message: 1. **Don't worry about specific machines** -- the scheduler handles placement 2. **Describe what your job needs** -- resources, time, constraints 3. **Default settings work for most cases** -- you only need to specify extras when you have special requirements

Exercises

1. Identify your workload type

Look at the workload types discussed in this module (batch, interactive, parallel, GPU-accelerated). Think about the computational work you do most often. Which category does it fall into? Would a single job cover it, or would you need multiple jobs? Write down your workload type and a one-sentence description.

Hint / Solution

There is no single right answer. Examples: - "My work is **batch** -- I run BLAST searches against a protein database that take 2-8 hours each." - "My work is **GPU-accelerated batch** -- I run CryoSPARC reconstruction jobs that need 1-2 GPUs for 12-24 hours." - "My work is a mix of **interactive** (testing parameters in a Jupyter session) and **batch** (running production simulations overnight)."

1. Describe the contract

Write out the "resource contract" for your typical job. What would you tell the scheduler you need? Consider: How many CPUs? How much memory? How long does it usually run? Does it need a GPU? Does it need access to specific software or data?

Hint / Solution

A good resource contract might look like: - **CPUs:** 4 cores (my tool is multithreaded but not massively parallel) - **Memory:** 16 GB (my datasets are ~10 GB and the tool needs overhead) - **Time:** 6 hours (most runs finish in 3-4 hours, but some take longer) - **GPU:** None - **Software:** RELION 4.0, loaded via environment modules - **Data:** Input files on `/scratch`, output to `/scratch/results` This is exactly the kind of information you will translate into Slurm directives in later modules.

References

• SchedMD: Slurm Overview
• SchedMD: Slurm Workload Manager
• Adapted from BioTeam SGE Training Materials (DRM Introduction)