Schrodinger Admin

<!-- audience: L4 --> <!-- deployment: all --> <!-- sge-source: none --> # Schrodinger Suite & Slurm -- Admin Setup ## Overview Schrodinger Suite is an integrated computational chemistry platform used in pharmaceutical drug discovery, materials science, and molecular modeling. It includes applications like Glide (docking), Desmond (molecular dynamics), FEP+ (free energy perturbation), and Prime (homology modeling). Schrodinger has its own job submission layer that sits **on top of** Slurm. Users submit jobs through Maestro (GUI) or command-line tools, and Schrodinger translates those into `sbatch` commands via its hosts file configuration. ### Two Job Management Architectures | | Job Control (Legacy) | Job Server (Modern) | |---|---|---| | **Config file** | `schrodinger.hosts` (flat text) | `hosts.yml` (YAML) | | **Config location** | `$SCHRODINGER/` installation dir | `<jobserver_dir>/config/` | | **Service** | None (direct SSH + sbatch) | `jobserverd` daemon | | **Backend** | Flat files | SQLite or PostgreSQL | | **Monitoring** | `$SCHRODINGER/jobcontrol` | `$SCHRODINGER/jsc` | | **Status** | Deprecated | Recommended for new installs | Job Server (`jobserverd`) is the recommended architecture for new deployments. It provides centralized job management, a REST API, and better scalability. This module covers both configurations since many sites still run Job Control. --- ## Slurm Prerequisites Before configuring Schrodinger, ensure your Slurm cluster meets these requirements: ### Shared Filesystem Schrodinger requires a shared filesystem accessible from the submit host and all compute nodes: - Software installation (`$SCHRODINGER`) must be accessible on all compute nodes - Scratch directory (`tmpdir`) can be shared or local (local is faster for I/O-heavy jobs) - Job output directories must be accessible from the submit host ### Network Connectivity - Compute nodes must reach the FLEXlm/SLM license server (TCP, typically port 27000+) - For Job Server: compute nodes must reach the jobserverd host (TCP 8029-8030) - Passwordless SSH between the submit host and compute nodes (for Job Control) ### Slurm Partitions Design partitions to match common Schrodinger workload types: ```ini # slurm.conf partition examples for Schrodinger PartitionName=cpu_driver Nodes=cpu[001-004] MaxTime=7-00:00:00 Default=NO PartitionName=cpu Nodes=cpu[001-064] MaxTime=7-00:00:00 Default=YES PartitionName=gpu Nodes=gpu[01-08] MaxTime=3-00:00:00 Default=NO ``` - **cpu_driver**: lightweight partition for workflow orchestration jobs (Glide grid generation, workflow drivers) - **cpu**: general compute for Glide docking, Prime, LigPrep, QikProp - **gpu**: GPU-accelerated jobs (Desmond MD, FEP+) --- ## Configuring the Hosts File (Job Control) The `schrodinger.hosts` file tells Schrodinger how to submit jobs to Slurm. It lives in the Schrodinger installation directory. ### Queue Directive The key directive is `Queue: SLURM2.1` -- this tells Schrodinger to use the Slurm submission scripts in `$SCHRODINGER/queues/SLURM2.1/`. ### Setting QPATH First, configure Schrodinger to find Slurm binaries: ```bash # Edit $SCHRODINGER/queues/SLURM2.1/config # Set QPATH to the Slurm bin directory QPATH=/opt/slurm/bin ``` ### Host Entry Examples ``` # schrodinger.hosts # Localhost entry (always required) name: localhost tmpdir: /scr # Workflow driver jobs (lightweight, orchestration) Name: driver Host: login01 Queue: SLURM2.1 Qargs: --partition=cpu_driver --nodes=1 --ntasks-per-node=%NPROC% Processors: 16 tmpdir: /scr # Standard CPU jobs (Glide, Prime, LigPrep) Name: cpu Host: login01 Queue: SLURM2.1 Qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% Processors: 2000 tmpdir: /scr # High-memory CPU jobs (large Prime, conformational search) Name: cpu_highmem Host: login01 Queue: SLURM2.1 Qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% --mem-per-cpu=8G Processors: 500 tmpdir: /scr # GPU jobs (Desmond, FEP+) Name: gpu Host: login01 Queue: SLURM2.1 Qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:%NPROC% Processors: 32 tmpdir: /scr ``` ### Key Configuration Details **`%NPROC%` variable:** Schrodinger substitutes this with the number of processors the user requests (`-NPROC`). When used in `--ntasks-per-node=%NPROC%`, Slurm allocates that many cores. When used in `--gres=gpu:%NPROC%`, Slurm allocates that many GPUs. **`Qargs` must be a single line:** No line breaks allowed in the Qargs value. **`Processors` setting:** The maximum number of processors a user can request for this host entry. **`Host` setting:** The host with access to the Slurm submission commands (typically a login node). --- ## Configuring hosts.yml (Job Server) For Job Server deployments, the YAML format (`hosts.yml`) replaces `schrodinger.hosts`: ```yaml # <jobserver_dir>/config/hosts.yml entries: - name: driver tmpdir: /scr qargs: --partition=cpu_driver --nodes=1 --ntasks-per-node=%NPROC% processors: 2000 processors_per_node: 4 env: - SCHRODINGER_MAX_LAUNCH_CONCURRENCY=4 - name: cpu tmpdir: /scr qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% processors: 2000 processors_per_node: 8 - name: cpu_highmem tmpdir: /scr qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% --mem-per-cpu 8G processors: 500 processors_per_node: 4 - name: gpu tmpdir: /scr qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:%NPROC% processors: 100 gpgpu: - index: 0 description: Tesla T4 - index: 1 description: Tesla T4 cuda_cores: 2560 ``` ### Key Differences from schrodinger.hosts | Feature | schrodinger.hosts | hosts.yml | |---------|------------------|-----------| | Format | Flat text, key: value | YAML | | `Queue:` directive | Required (`SLURM2.1`) | Not needed (Job Server knows) | | `Host:` directive | Required | Not needed (centralized) | | `processors_per_node` | Not available | Controls threaded job distribution | | `gpgpu` section | Not available | Declares GPU details for Maestro | | `env` section | Single `env:` line | YAML list of env vars | ### Validate and Reload ```bash # Validate YAML syntax sudo -u jobserver $SCHRODINGER/jsc admin check-hosts-config \ <jobserver_dir>/config/hosts.yml # Reload hosts config without restarting Job Server sudo -u jobserver $SCHRODINGER/jsc admin reload-hosts <hostname> ``` ### Converting from schrodinger.hosts to hosts.yml ```bash sudo -u jobserver $SCHRODINGER/jsc admin convert-schrodinger-hosts \ --job-server-dir=<jobserver_dir> \ --schrod=<schrodinger_installation_dir> ``` --- ## GPU Configuration ### Slurm-Side GPU Setup Ensure GRES is configured for GPU nodes: ```ini # gres.conf NodeName=gpu[01-08] Name=gpu Type=a100 File=/dev/nvidia[0-3] ``` ```ini # slurm.conf GresTypes=gpu ``` ### Schrodinger GPU Host Entry For GPU jobs (Desmond, FEP+), the `--gres=gpu:%NPROC%` in Qargs requests GPUs through Slurm: ``` # For typed GPUs (recommended) Qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:a100:%NPROC% # For generic GPUs Qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:%NPROC% ``` ### Desmond Multi-GPU Configuration Desmond (MD) can use multiple GPUs on a single node: ``` Name: desmond_gpu Host: login01 Queue: SLURM2.1 Qargs: --partition=gpu --nodes=1 --ntasks-per-node=1 --gres=gpu:4 --cpus-per-task=16 Processors: 8 tmpdir: /scr ``` --- ## OpenMPI for Distributed Jobs Some Schrodinger applications (Desmond, Quantum ESPRESSO) support multi-node MPI execution: ### Single-Node MPI ``` Qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% ``` ### Multi-Node MPI ``` Qargs: --partition=cpu --ntasks=%NPROC% ``` The difference: `--nodes=1` constrains to one node, while omitting it allows Slurm to spread tasks across multiple nodes. --- ## License Management ### Licensing Overview: SLM Replaces FlexNet As of the **2025-1 release**, Schrodinger has **discontinued FlexNet (FLEXlm) licensing**. All current releases require **Schrodinger License Manager (SLM)** licenses. If your organization is still running FlexNet-based licenses, you must transition to SLM before upgrading to any 2025-1 or later release. | | FlexNet (FLEXlm) | Schrodinger License Manager (SLM) | |---|---|---| | **Status** | **Discontinued** (no longer available) | **Current** (required for 2025-1+) | | **Server process** | `lmgrd` + vendor daemon | `licserverd` | | **Default port** | 27008, 53000 | 53000 (configurable) | | **Client config** | License file or `LM_LICENSE_FILE` | Server identifier file | | **Hosting options** | Self-hosted only | Self-hosted or **SLM Cloud** (Schrodinger-hosted) | | **License tool** | `lmutil` | `lictool` | ### SLM Deployment Options **Option 1: Self-Hosted SLM Server** You run `licserverd` on your own infrastructure (on-prem server or cloud instance). This is the direct replacement for a self-hosted FlexNet server. ```bash # Quick install on Linux sudo $SCHRODINGER/licadmin quick-slm-setup install -lickey <slm_lickey> # Default port: 53000 # Install location: /opt/schrodinger/licserver ``` **Option 2: SLM Cloud (Schrodinger-Hosted)** Schrodinger offers **SLM Cloud** as an optional paid service where Schrodinger hosts and manages the license server on their infrastructure (typically a GCP instance in a region near the customer). Schrodinger maintains the server, updates licenses, and troubleshoots server-side issues. **SLM Cloud requirements:** 1. **Static exit IP addresses:** Your organization must have well-defined static public/exit IP addresses. These are provided to Schrodinger (in CIDR notation) for safelisting. Personal VPNs and home ISP addresses are not supported. 2. **Firewall configuration:** All machines where Schrodinger license checkouts occur (compute nodes, login nodes, workstations) must be able to make outbound TCP connections to port 53000 of the Schrodinger-hosted license server. 3. **IP safelisting:** Send your exit IP addresses to `help@schrodinger.com` for safelisting on the Schrodinger-hosted server. **Determining your exit IP addresses:** ```bash # From compute nodes or login nodes curl ipinfo.io/ip # Or in a browser on the machine # https://checkip.amazonaws.com ``` > **ParallelCluster Note:** On ParallelCluster, compute nodes access the internet via a NAT gateway. The NAT gateway's Elastic IP is the exit IP address that must be safelisted with Schrodinger. If you use multiple AZs with separate NAT gateways, all their EIPs must be safelisted. > **PCS Note:** PCS compute nodes similarly route through NAT. Identify the NAT gateway EIPs in your VPC and provide those to Schrodinger for safelisting. Security groups must allow outbound TCP 53000 to the Schrodinger-hosted server. > **On-Prem Note:** On-prem clusters may route through a corporate firewall, proxy, or VPN. Work with your network team to identify the public exit IP(s) for cluster traffic and ensure that the firewall allows outbound TCP 53000 to the Schrodinger SLM Cloud server. If your organization uses Zscaler or similar web proxies, ensure direct connectivity (not proxied) to the SLM server. **Testing SLM Cloud access:** ```bash # Test license server connectivity $SCHRODINGER/run lictool test-server -f MMLIBS <hostname>:53000 # Should report "success" # List available license features $SCHRODINGER/run lictool status --licsrv-hostport <hostname>:53000 ``` ### Transitioning from FlexNet to SLM If migrating from an existing FlexNet installation: 1. **Run both in parallel** (recommended): Use a different port for SLM (e.g., 53001) on the same server machine 2. **Update firewall rules** to allow the new SLM port 3. **Reconfigure all client machines** -- FlexNet client configurations do NOT work with SLM 4. **Update the Slurm license sensor** -- set `SCHRODINGER_LICENSING_NAMESPACE = "SLM_"` to distinguish SLM licenses from FlexNet in the Slurm accounting database (or set to empty string if FlexNet is fully retired) ### Slurm License-Aware Scheduling On a Slurm cluster, license availability should be integrated with Slurm's accounting to prevent jobs from starting when licenses are unavailable. Slurm can hold jobs in the queue until the required licenses are free, then dispatch them automatically. > **Best Practice:** Slurm accounting (slurmdbd) **must** be enabled for license-aware scheduling to function. Without accounting, none of the license integration described below will work -- jobs will start regardless of license availability, leading to license checkout failures, wasted compute hours, and frustrated users. > > If your cluster does not yet have accounting enabled, see the deployment modules ([on-premise](../deployment/on-premise.md), [ParallelCluster](../deployment/aws-parallelcluster.md), [PCS](../deployment/aws-pcs.md)) for setup instructions. Accounting is also required for: > > - **Fairshare scheduling** -- prevents heavy Schrodinger users from starving other workloads > - **QOS enforcement** -- set limits on concurrent Schrodinger jobs per user or group > - **Job history and reporting** (`sacct`, `sreport`) -- track license and compute usage over time > - **Account-based chargeback** -- attribute compute costs to projects or grants > - **Resource limit enforcement** (MaxJobs, GrpTRES) -- cap total consumption per user ### Two Approaches: Local vs. Remote License Checking Choose the appropriate approach based on whether Schrodinger licenses are consumed only on the HPC cluster or also by external machines (laptops, workstations). | Scenario | Approach | Sensor Required? | |----------|---------|-----------------| | **Schrodinger used only on the Slurm cluster** | Local license checking | **No** -- static `Licenses=` lines in slurm.conf | | **Schrodinger also used on workstations, laptops, other clusters** | Remote license checking | **Yes** -- license sensor polls and updates counts dynamically | ### Local License Checking (HPC-Only) If all Schrodinger license checkouts happen through Slurm (no external users on workstations), you can use **static license definitions** in `slurm.conf`. This is the simpler approach -- no sensor script is needed. **Step 1: Generate Slurm-Compatible License Lines** Schrodinger provides the `licutil` utility that queries your license server and emits `Licenses=` lines ready to paste into `slurm.conf`: ```bash # Generate license config lines for slurm.conf $SCHRODINGER/utilities/licutil -slurmconf ``` This outputs lines like: ```ini Licenses=MMLIBS:100,GLIDE_SP:50,GLIDE_XP:25,DESMOND_GPGPU:16,FEP_GPGPU:8 ``` **Step 2: The Prefix Question** By default, `licutil` adds an `SLM_` prefix to distinguish SLM licenses from legacy FlexNet: ```bash # Default behavior (prefix = "SLM") $SCHRODINGER/utilities/licutil -slurmconf # Output: Licenses=SLM_MMLIBS:100,SLM_GLIDE_SP:50,... # Explicit prefix $SCHRODINGER/utilities/licutil -slurmconf -prefix SLM # Output: Licenses=SLM_MMLIBS:100,SLM_GLIDE_SP:50,... # No prefix (bare feature names) $SCHRODINGER/utilities/licutil -slurmconf -prefix "" # Output: Licenses=MMLIBS:100,GLIDE_SP:50,... ``` **Prefix vs. no prefix -- pros and cons:** | Approach | Pros | Cons | |----------|------|------| | **With prefix (`SLM_`)** | Distinguishes SLM from FlexNet licenses during migration; prevents name collisions if other software uses the same feature names | Longer names in Slurm output; must match `schrodinger_licensing_namespace` in `jobserver.yml` | | **No prefix (`""`)** | Simpler, cleaner names; recommended when FlexNet is fully retired | Cannot distinguish SLM from FlexNet features during migration; potential name collisions with non-Schrodinger licenses | > **Recommendation:** If you have fully transitioned to SLM and no longer run FlexNet, use `-prefix ""` for simplicity. The prefix must match the `schrodinger_licensing_namespace` setting in `jobserver.yml` (set it to `""` there too). If you are mid-migration or share a Slurm accounting database with clusters that still use FlexNet, keep the `SLM_` prefix. **Step 3: Add to slurm.conf** ```ini # slurm.conf Licenses=MMLIBS:100,GLIDE_SP:50,GLIDE_XP:25,DESMOND_GPGPU:16,FEP_GPGPU:8 # Required for license-aware job requeuing JobRequeue=1 RequeueExit=99 ``` Restart slurmctld after adding: ```bash systemctl restart slurmctld ``` **Step 4: Verify** ```bash scontrol show lic # LicenseName=MMLIBS # Total=100 Used=0 Free=100 Remote=no # LicenseName=GLIDE_SP # Total=50 Used=0 Free=50 Remote=no ``` > **Note:** If your license entitlements change (new features added, counts updated), you must re-run `licutil -slurmconf`, update `slurm.conf`, and restart slurmctld. There is no automatic discovery of license changes in local mode. ### Remote License Checking (Mixed Environment) If Schrodinger licenses are consumed both by Slurm jobs and by external users (workstations, laptops via VPN, other clusters), the static `Licenses=` approach will be inaccurate -- Slurm won't know about external checkouts. In this case, use the **license sensor** to dynamically track actual license availability. **How it works:** The sensor script (`slurm-license-sensor`) polls the SLM license server every 30 seconds, compares current checkouts against the total, and updates the Slurm accounting database. The `CLUSTER_DNS_NAME_REGEX` setting tells the sensor which checkouts are from Slurm nodes (tracked by Slurm) vs. external machines (which reduce the available count). **Step 1: Configure the sensor** ```bash # Edit $SCHRODINGER/utilities/slurm-license-sensor CLUSTER_DNS_NAME_REGEX="compute-.*\\.cluster\\.local" # Match your compute node hostnames SLURM_CLUSTERS=["mycluster"] # Your Slurm cluster name(s) SLURM_INSTALLDIR="/opt/slurm" SCHRODINGER_LICENSING_NAMESPACE="SLM_" # Match your prefix choice (or "") ``` **Step 2: Create a systemd timer** ```ini # /etc/systemd/system/slurm-license-sensor.service [Unit] Description=Schrodinger license sensor for Slurm After=slurmctld.service slurmdbd.service [Service] Type=simple User=slurm ExecStart=/opt/schrodinger2026-1/run /opt/schrodinger2026-1/utilities/slurm-license-sensor [Install] WantedBy=multi-user.target ``` ```ini # /etc/systemd/system/slurm-license-sensor.timer [Unit] Description=Schrodinger license sensor for Slurm [Timer] OnCalendar=*:*:0/30 AccuracySec=1s [Install] WantedBy=timers.target ``` ```bash chmod +x $SCHRODINGER/utilities/slurm-license-sensor systemctl enable --now slurm-license-sensor.timer ``` The sensor automatically registers license resources in the Slurm accounting database: ```bash # The sensor creates entries like: sacctmgr -i add resource name=MMLIBS count=100 \ server=schrodinger_mycluster cluster=mycluster \ servertype=flexlm percentallowed=100 ``` For multi-cluster environments, each cluster gets its own server namespace with `percentallowed=100`, avoiding the need to pre-allocate fixed percentages. **Step 3: Verify** ```bash scontrol show lic | grep -A1 mmlibs # LicenseName=MMLIBS # Total=100 Used=0 Free=100 Remote=yes ``` Note `Remote=yes` -- this indicates the license count is managed by the accounting database (sensor-driven) rather than a static slurm.conf entry. ### Caveats and Limitations - **Update latency:** The sensor runs every 30 seconds. Brief race conditions are possible where a job starts before an external checkout is registered. - **Suite licenses (PACKAGE):** If features A and B are both part of a license package, checking out A does not automatically update B's count. Suite licenses are not perfectly tracked. - **Multi-server aggregation:** The sensor sums availability across multiple license servers, but a single checkout cannot span servers. - **Local mode is static:** If using `Licenses=` in slurm.conf (no sensor), license counts do not update when entitlements change -- you must manually re-run `licutil` and restart slurmctld. ### Testing License Integration ```bash # Check license status in Slurm scontrol show lic | grep -A1 mmlibs # Submit a test job that checks out licenses $SCHRODINGER/testapp -HOST cpu -t 300 -l MMLIBS:4 -j testapp.mmlibs_4 # Verify license count updated while job runs scontrol show lic | grep -A1 mmlibs # Total=<N> Used=4 Free=<N-4> Remote=yes ``` ### SCHRODINGER_LICENSE_RETRY Set this environment variable in host entries to handle transient license server issues: ``` # In schrodinger.hosts env: SCHRODINGER_LICENSE_RETRY=5 # In hosts.yml env: - SCHRODINGER_LICENSE_RETRY=5 ``` This retries license checkout up to 5 times before failing. Useful for SLM Cloud where network latency may cause occasional transient failures. --- ## Job Server Setup ### Installation ```bash # Run the setup utility $SCHRODINGER/utilities/setup_jobserver -setup_jobserver ``` ### Configure jobserverd for Slurm Ensure Slurm binaries are in the PATH for the jobserver user: ```bash # /etc/profile.d/slurm.sh export PATH=$PATH:/opt/slurm/bin/:/opt/slurm/sbin/ ``` ### Start and Verify ```bash # Test manually sudo -u jobserver /bin/bash -c \ "source /etc/profile.d/slurm.sh && <jobserver_dir>/bin/jobserverd --debug --dir <jobserver_dir>" # Create systemd service cat > /etc/systemd/system/jobserverd.service <<EOF [Unit] Description=Schrodinger Job Server After=network.target [Service] Type=simple User=jobserver ExecStart=/bin/bash -c "source /etc/profile.d/slurm.sh && <jobserver_dir>/bin/jobserverd --dir <jobserver_dir>" Restart=always [Install] WantedBy=multi-user.target EOF systemctl enable --now jobserverd ``` ### Job Server Ports | Port | Purpose | |------|---------| | 8029 | File server (job data transfer) | | 8030 | Job server API | Ensure these ports are accessible from all compute nodes and user workstations. --- ## Testing & Validation ### installation_check Run Schrodinger's built-in validation: ```bash $SCHRODINGER/installation_check ``` This tests license connectivity, library loading, and basic functionality. ### Test Queue Submission ```bash # Submit a test job to each host entry $SCHRODINGER/testapp -HOST cpu -t 60 -j test_cpu $SCHRODINGER/testapp -HOST gpu -t 60 -j test_gpu # Monitor with Job Control $SCHRODINGER/jobcontrol -list # Or with Job Server $SCHRODINGER/jsc list ``` ### Verify Slurm Integration ```bash # Check that the Schrodinger job appears in Slurm squeue -u $USER # After completion, check accounting sacct -j <JOBID> --format=JobID,JobName,Elapsed,MaxRSS,ExitCode ``` --- ## Deployment Callouts > **ParallelCluster Note:** On ParallelCluster, the Schrodinger installation should be on shared storage (FSx Lustre or EFS) accessible from all compute nodes. The `tmpdir` should point to local NVMe instance storage or a fast shared filesystem. Partition names follow the ParallelCluster naming convention (e.g., `cpu-st-c5-4xl`), so update Qargs accordingly. License server access requires VPC security group rules allowing outbound TCP to the license server. > **PCS Note:** On PCS, install Schrodinger in a custom AMI or on shared storage (EFS/FSx). Configure host entries in your custom AMI's Schrodinger installation. PCS queue names map to your compute node group names. > **On-Prem Note:** On-prem clusters have the most flexibility. You can mount the Schrodinger installation via NFS, use local scratch for tmpdir, and configure license servers on the local network. For multi-cluster environments with separate Job Servers, ensure host entry names are unique across clusters.

Related Modules

• Schrodinger User Guide -- end-user perspective on Schrodinger + Slurm
• Resource Management -- GRES and license configuration in Slurm
• On-Premise Deployment -- cluster setup for Schrodinger workloads
• AWS ParallelCluster -- cloud deployment considerations

References

• Schrodinger: Setup for Slurm Queueing Systems
• Schrodinger: OpenMPI Setup for Slurm
• Schrodinger: GPU Configuration for Slurm
• Schrodinger: The Hosts File
• Schrodinger: Job Server Administration
• Schrodinger: License Checking for Slurm
• SchedMD: Slurm Licenses Guide
• SchedMD: GRES Configuration