Schrodinger Admin

# Schrodinger Suite & Slurm -- Admin Setup

## Overview

Schrodinger Suite is an integrated computational chemistry platform used in pharmaceutical drug discovery, materials science, and molecular modeling. It includes applications like Glide (docking), Desmond (molecular dynamics), FEP+ (free energy perturbation), and Prime (homology modeling).

Schrodinger has its own job submission layer that sits **on top of** Slurm. Users submit jobs through Maestro (GUI) or command-line tools, and Schrodinger translates those into `sbatch` commands via its hosts file configuration.

### Two Job Management Architectures

| | Job Control (Legacy) | Job Server (Modern) |
|---|---|---|
| **Config file** | `schrodinger.hosts` (flat text) | `hosts.yml` (YAML) |
| **Config location** | `$SCHRODINGER/` installation dir | `<jobserver_dir>/config/` |
| **Service** | None (direct SSH + sbatch) | `jobserverd` daemon |
| **Backend** | Flat files | SQLite or PostgreSQL |
| **Monitoring** | `$SCHRODINGER/jobcontrol` | `$SCHRODINGER/jsc` |
| **Status** | Deprecated | Recommended for new installs |

Job Server (`jobserverd`) is the recommended architecture for new deployments. It provides centralized job management, a REST API, and better scalability. This module covers both configurations since many sites still run Job Control.

---

## Slurm Prerequisites

Before configuring Schrodinger, ensure your Slurm cluster meets these requirements:

---

## Shared Filesystem

Schrodinger requires a shared filesystem accessible from the submit host and all compute nodes:

- Software installation (`$SCHRODINGER`) must be accessible on all compute nodes
- Scratch directory (`tmpdir`) can be shared or local (local is faster for I/O-heavy jobs)
- Job output directories must be accessible from the submit host

## Network Connectivity

- Compute nodes must reach the FLEXlm/SLM license server (TCP, typically port 27000+)
- For Job Server: compute nodes must reach the jobserverd host (TCP 8029-8030)
- Passwordless SSH between the submit host and compute nodes (for Job Control)

## Slurm Partitions

Design partitions to match common Schrodinger workload types:

```ini
# slurm.conf partition examples for Schrodinger
PartitionName=cpu_driver Nodes=cpu[001-004] MaxTime=7-00:00:00 Default=NO
PartitionName=cpu Nodes=cpu[001-064] MaxTime=7-00:00:00 Default=YES
PartitionName=gpu Nodes=gpu[01-08] MaxTime=3-00:00:00 Default=NO
```

- **cpu_driver**: lightweight partition for workflow orchestration jobs (Glide grid generation, workflow drivers)
- **cpu**: general compute for Glide docking, Prime, LigPrep, QikProp
- **gpu**: GPU-accelerated jobs (Desmond MD, FEP+)

---

## Configuring the Hosts File (Job Control)

The `schrodinger.hosts` file tells Schrodinger how to submit jobs to Slurm. It lives in the Schrodinger installation directory.

### Queue Directive

The key directive is `Queue: SLURM2.1` -- this tells Schrodinger to use the Slurm submission scripts in `$SCHRODINGER/queues/SLURM2.1/`.

### Setting QPATH

First, configure Schrodinger to find Slurm binaries:

```bash
# Edit $SCHRODINGER/queues/SLURM2.1/config
# Set QPATH to the Slurm bin directory
QPATH=/opt/slurm/bin
```

---

## Host Entry Examples

```
# schrodinger.hosts

# Localhost entry (always required by the hosts file -- but never route compute-class
# products to it: "localhost" means the login node, outside Slurm's control)
name:         localhost
tmpdir:       /scr

# Workflow driver jobs (lightweight, orchestration)
Name:         driver
Host:         login01
Queue:        SLURM2.1
Qargs:        --partition=cpu_driver --nodes=1 --ntasks-per-node=%NPROC%
Processors:   16
tmpdir:       /scr

# Standard CPU jobs (Glide, Prime, LigPrep)
Name:         cpu
Host:         login01
Queue:        SLURM2.1
Qargs:        --partition=cpu --nodes=1 --ntasks-per-node=%NPROC%
Processors:   2000
tmpdir:       /scr

# High-memory CPU jobs (large Prime, conformational search)
Name:         cpu_highmem
Host:         login01
Queue:        SLURM2.1
Qargs:        --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% --mem-per-cpu=8G
Processors:   500
tmpdir:       /scr

# GPU jobs (Desmond, FEP+)
Name:         gpu
Host:         login01
Queue:        SLURM2.1
Qargs:        --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:%NPROC%
Processors:   32
tmpdir:       /scr
```

## Key Configuration Details

**`%NPROC%` variable:** Schrodinger substitutes this with the number of processors the user requests (`-NPROC`). When used in `--ntasks-per-node=%NPROC%`, Slurm allocates that many cores. When used in `--gres=gpu:%NPROC%`, Slurm allocates that many GPUs.

**`Qargs` must be a single line:** No line breaks allowed in the Qargs value.

**`Processors` setting:** The maximum number of processors a user can request for this host entry.

**`Host` setting:** The host with access to the Slurm submission commands (typically a login node).

---

## Configuring hosts.yml (Job Server)

For Job Server deployments, the YAML format (`hosts.yml`) replaces `schrodinger.hosts`:

```yaml
# <jobserver_dir>/config/hosts.yml
entries:

- name: driver
    tmpdir: /scr
    qargs: --partition=cpu_driver --nodes=1 --ntasks-per-node=%NPROC%
    processors: 2000
    processors_per_node: 4
    env:
      - SCHRODINGER_MAX_LAUNCH_CONCURRENCY=4

- name: cpu
    tmpdir: /scr
    qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC%
    processors: 2000
    processors_per_node: 8

- name: cpu_highmem
    tmpdir: /scr
    qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC% --mem-per-cpu 8G
    processors: 500
    processors_per_node: 4

- name: gpu
    tmpdir: /scr
    qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:%NPROC%
    processors: 100
    gpgpu:
      - index: 0
        description: Tesla T4
      - index: 1
        description: Tesla T4
    cuda_cores: 2560
```

---

## Key Differences from schrodinger.hosts

| Feature | schrodinger.hosts | hosts.yml |
|---------|------------------|-----------|
| Format | Flat text, key: value | YAML |
| `Queue:` directive | Required (`SLURM2.1`) | Not needed (Job Server knows) |
| `Host:` directive | Required | Not needed (centralized) |
| `processors_per_node` | Not available | Controls threaded job distribution |
| `gpgpu` section | Not available | Declares GPU details for Maestro |
| `env` section | Single `env:` line | YAML list of env vars |

---

## Validate and Reload

```bash
# Validate YAML syntax
sudo -u jobserver $SCHRODINGER/jsc admin check-hosts-config \
    <jobserver_dir>/config/hosts.yml

# Reload hosts config without restarting Job Server
sudo -u jobserver $SCHRODINGER/jsc admin reload-hosts <hostname>
```

## Converting from schrodinger.hosts to hosts.yml

```bash
sudo -u jobserver $SCHRODINGER/jsc admin convert-schrodinger-hosts \
    --job-server-dir=<jobserver_dir> \
    --schrod=<schrodinger_installation_dir>
```

---

## Operating the Job Server on Slurm

Two records exist for every job: Slurm's (what Slurm was told) and the Job Server's (what the driver was told). Most "stuck job", wrong-license and missing-history tickets are the wrong record read first.

**Map a Slurm job to a `jsc` id.** The Job Server writes its UUID into the Slurm job comment:

```bash
squeue -a -o '%i %k'                      # JOBID and Comment (%k)
scontrol -a show job <slurm_jobid> | grep -oE 'SchrodingerJobId=[^ ,]*'
# Comment=SchrodingerJobId=<uuid>  ->  jsc info / tail-file / postmortem <uuid>
```

Direct `sbatch` jobs have no such comment. `jsc info` on another user's job is usually denied, so start from Slurm.

**`localhost` routes compute to the login node.** A `localhost` host entry runs the job where the user launched it. Slurm never sees it: no accounting, no limits, no license accounting through the scheduler. Count the share from the job DB (`$SCHRODINGER/jsc list | wc -l`), not `squeue`, and drop `localhost` for compute-class products.

**`cuda_cores` must match the hardware.** It sets how many GPU license tokens a host entry reserves. A wrong value draws the wrong token count and shows as failed jobs concentrated on one entry.

```bash
nvidia-smi --query-gpu=name,count --format=csv,noheader
scontrol show node <gpu_node> | grep -E 'Gres=|CfgTRES'
```

---

## Job Server: History, Enrolment, Versions

**The job DB prunes itself** (`job_server` section of `jobserver.yml`, defaults):

```yaml
failed_job_cleanup_days:    7    # Failed and Canceled
done_job_cleanup_days:     14    # Completed/Stopped, files downloaded
abandoned_job_cleanup_days: 28   # Completed/Stopped, files never downloaded
```

`jsc list` is a snapshot of the last 1-4 weeks, and failed jobs vanish first, so failure rates computed from it are biased. Use `sacct` for history.

**A fresh account needs enrolment.** `jsc server-info` fails on an account that never used the Job Server. That is not an outage; run once per user per submission host:

```bash
$SCHRODINGER/jsc cert get <jobserver_host>[:<port>]
```

**One module per version, no `default` symlink.** If `/etc/profile.d/schrodinger.sh` exports a path ending in `default`, nothing records which release a job used and an upgrade silently changes new jobs. Install releases side by side, ship one environment module per version, and log `module list` in the job.

**The launch directory must exist on the compute nodes.** If it exists only on the front end, the MD engine falls back to a node-local `-tmpdir` and the write-back fails at the end of a long job. Compare `ls -ld <launch_dir>` on the front end with `srun -w <node> ls -ld <launch_dir>`.

---

## Job Server: Licenses and "Stuck" Jobs

- **`Remote=yes` binds only to Job Server submissions.** A direct `sbatch` shows `(null)` in `squeue -o %W` and consumes licenses Slurm does not count (`sacctmgr -n show resource`, `scontrol show licenses`).
- **License daemons on the controller go down with it.** `lmgrd` or `rlm` found on the controller or a login node means a reboot or rebuild takes every application license out with the scheduler. Use a dedicated license host.
- **A GPU at about 100 % utilization near its power limit is computing, not stuck.** A growing `etimes` alone proves nothing.

```bash
ps -eo comm,args | grep -E 'lmgrd|rlm' | grep -v grep
nvidia-smi --query-gpu=index,utilization.gpu,power.draw,power.limit --format=csv
```

| Symptom | First check |
|---|---|
| `jsc server-info` fails for a new user | `jsc cert get` enrolment missing, not an outage |
| Job "stuck", process age growing | GPU utilization and `power.draw` vs `power.limit`; near 100 % at near-TDP = computing (a hang would more likely show 0 % at idle power -- inference, not vendor-documented) |
| Licenses drawn do not match GPUs used | `cuda_cores` and GPU model in the host entry vs `nvidia-smi` |

---

## GPU Configuration

### Slurm-Side GPU Setup

Ensure GRES is configured for GPU nodes:

```ini
# gres.conf
NodeName=gpu[01-08] Name=gpu Type=a100 File=/dev/nvidia[0-3]
```

```ini
# slurm.conf
GresTypes=gpu
```

### Schrodinger GPU Host Entry

For GPU jobs (Desmond, FEP+), the `--gres=gpu:%NPROC%` in Qargs requests GPUs through Slurm:

```
# For typed GPUs (recommended)
Qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:a100:%NPROC%

# For generic GPUs
Qargs: --partition=gpu --nodes=1 --ntasks-per-node=%NPROC% --gres=gpu:%NPROC%
```

### Desmond Multi-GPU Configuration

Desmond (MD) can use multiple GPUs on a single node:

```
Name:         desmond_gpu
Host:         login01
Queue:        SLURM2.1
Qargs:        --partition=gpu --nodes=1 --ntasks-per-node=1 --gres=gpu:4 --cpus-per-task=16
Processors:   8
tmpdir:       /scr
```

---

## OpenMPI for Distributed Jobs

Some Schrodinger applications (Desmond, Quantum ESPRESSO) support multi-node MPI execution:

### Single-Node MPI

```
Qargs: --partition=cpu --nodes=1 --ntasks-per-node=%NPROC%
```

### Multi-Node MPI

```
Qargs: --partition=cpu --ntasks=%NPROC%
```

The difference: `--nodes=1` constrains to one node, while omitting it allows Slurm to spread tasks across multiple nodes.

---

## License Management

## Licensing Overview: SLM Replaces FlexNet

As of the **2025-1 release**, Schrodinger has **discontinued FlexNet (FLEXlm) licensing**. All current releases require **Schrodinger License Manager (SLM)** licenses. If your organization is still running FlexNet-based licenses, you must transition to SLM before upgrading to any 2025-1 or later release.

| | FlexNet (FLEXlm) | Schrodinger License Manager (SLM) |
|---|---|---|
| **Status** | **Discontinued** (no longer available) | **Current** (required for 2025-1+) |
| **Server process** | `lmgrd` + vendor daemon | `licserverd` |
| **Default port** | 27008, 53000 | 53000 (configurable) |
| **Client config** | License file or `LM_LICENSE_FILE` | Server identifier file |
| **Hosting options** | Self-hosted only | Self-hosted or **SLM Cloud** (Schrodinger-hosted) |
| **License tool** | `lmutil` | `lictool` |

---

## SLM Deployment Options

**Option 1: Self-Hosted SLM Server**

You run `licserverd` on your own infrastructure (on-prem server or cloud instance). This is the direct replacement for a self-hosted FlexNet server.

```bash
# Quick install on Linux
sudo $SCHRODINGER/licadmin quick-slm-setup install -lickey <slm_lickey>

# Default port: 53000
# Install location: /opt/schrodinger/licserver
```

**Option 2: SLM Cloud (Schrodinger-Hosted)**

Schrodinger offers **SLM Cloud** as an optional paid service where Schrodinger hosts and manages the license server on their infrastructure (typically a GCP instance in a region near the customer). Schrodinger maintains the server, updates licenses, and troubleshoots server-side issues.

**SLM Cloud requirements:**

1. **Static exit IP addresses:** Your organization must have well-defined static public/exit IP addresses. These are provided to Schrodinger (in CIDR notation) for safelisting. Personal VPNs and home ISP addresses are not supported.

2. **Firewall configuration:** All machines where Schrodinger license checkouts occur (compute nodes, login nodes, workstations) must be able to make outbound TCP connections to port 53000 of the Schrodinger-hosted license server.

3. **IP safelisting:** Send your exit IP addresses to `help@schrodinger.com` for safelisting on the Schrodinger-hosted server.

**Determining your exit IP addresses:**
```bash
# From compute nodes or login nodes
curl ipinfo.io/ip

# Or in a browser on the machine
# https://checkip.amazonaws.com
```

> **ParallelCluster Note:** On ParallelCluster, compute nodes access the internet via a NAT gateway. The NAT gateway's Elastic IP is the exit IP address that must be safelisted with Schrodinger. If you use multiple AZs with separate NAT gateways, all their EIPs must be safelisted.

> **PCS Note:** PCS compute nodes similarly route through NAT. Identify the NAT gateway EIPs in your VPC and provide those to Schrodinger for safelisting. Security groups must allow outbound TCP 53000 to the Schrodinger-hosted server.

> **On-Prem Note:** On-prem clusters may route through a corporate firewall, proxy, or VPN. Work with your network team to identify the public exit IP(s) for cluster traffic and ensure that the firewall allows outbound TCP 53000 to the Schrodinger SLM Cloud server. If your organization uses Zscaler or similar web proxies, ensure direct connectivity (not proxied) to the SLM server.

**Testing SLM Cloud access:**
```bash
# Test license server connectivity
$SCHRODINGER/run lictool test-server -f MMLIBS <hostname>:53000
# Should report "success"

# List available license features
$SCHRODINGER/run lictool status --licsrv-hostport <hostname>:53000
```

---

## Transitioning from FlexNet to SLM

If migrating from an existing FlexNet installation:

1. **Run both in parallel** (recommended): Use a different port for SLM (e.g., 53001) on the same server machine
2. **Update firewall rules** to allow the new SLM port
3. **Reconfigure all client machines** -- FlexNet client configurations do NOT work with SLM
4. **Update the Slurm license sensor** -- set `SCHRODINGER_LICENSING_NAMESPACE = "SLM_"` to distinguish SLM licenses from FlexNet in the Slurm accounting database (or set to empty string if FlexNet is fully retired)

---

## Slurm License-Aware Scheduling

On a Slurm cluster, license availability should be integrated with Slurm's accounting to prevent jobs from starting when licenses are unavailable. Slurm can hold jobs in the queue until the required licenses are free, then dispatch them automatically.

> **Best Practice:** Slurm accounting (slurmdbd) **must** be enabled for license-aware scheduling to function. Without accounting, none of the license integration described below will work -- jobs will start regardless of license availability, leading to license checkout failures, wasted compute hours, and frustrated users.
>
> If your cluster does not yet have accounting enabled, see the deployment modules ([on-premise](../deployment/on-premise.md), [ParallelCluster](../deployment/aws-parallelcluster.md), [PCS](../deployment/aws-pcs.md)) for setup instructions. Accounting is also required for:
>
> - **Fairshare scheduling** -- prevents heavy Schrodinger users from starving other workloads
> - **QOS enforcement** -- set limits on concurrent Schrodinger jobs per user or group
> - **Job history and reporting** (`sacct`, `sreport`) -- track license and compute usage over time
> - **Account-based chargeback** -- attribute compute costs to projects or grants
> - **Resource limit enforcement** (MaxJobs, GrpTRES) -- cap total consumption per user

---

## Two Approaches: Local vs. Remote License Checking

Choose the appropriate approach based on whether Schrodinger licenses are consumed only on the HPC cluster or also by external machines (laptops, workstations).

| Scenario | Approach | Sensor Required? |
|----------|---------|-----------------|
| **Schrodinger used only on the Slurm cluster** | Local license checking | **No** -- static `Licenses=` lines in slurm.conf |
| **Schrodinger also used on workstations, laptops, other clusters** | Remote license checking | **Yes** -- license sensor polls and updates counts dynamically |

---

## Local License Checking (HPC-Only)

If all Schrodinger license checkouts happen through Slurm (no external users on workstations), you can use **static license definitions** in `slurm.conf`. This is the simpler approach -- no sensor script is needed.

**Step 1: Generate Slurm-Compatible License Lines**

Schrodinger provides the `licutil` utility that queries your license server and emits `Licenses=` lines ready to paste into `slurm.conf`:

```bash
# Generate license config lines for slurm.conf
$SCHRODINGER/utilities/licutil -slurmconf
```

This outputs lines like:

```ini
Licenses=MMLIBS:100,GLIDE_SP:50,GLIDE_XP:25,DESMOND_GPGPU:16,FEP_GPGPU:8
```

**Step 2: The Prefix Question**

By default, `licutil` adds an `SLM_` prefix to distinguish SLM licenses from legacy FlexNet:

```bash
# Default behavior (prefix = "SLM")
$SCHRODINGER/utilities/licutil -slurmconf
# Output: Licenses=SLM_MMLIBS:100,SLM_GLIDE_SP:50,...

# Explicit prefix
$SCHRODINGER/utilities/licutil -slurmconf -prefix SLM
# Output: Licenses=SLM_MMLIBS:100,SLM_GLIDE_SP:50,...

# No prefix (bare feature names)
$SCHRODINGER/utilities/licutil -slurmconf -prefix ""
# Output: Licenses=MMLIBS:100,GLIDE_SP:50,...
```

**Prefix vs. no prefix -- pros and cons:**

| Approach | Pros | Cons |
|----------|------|------|
| **With prefix (`SLM_`)** | Distinguishes SLM from FlexNet licenses during migration; prevents name collisions if other software uses the same feature names | Longer names in Slurm output; must match `schrodinger_licensing_namespace` in `jobserver.yml` |
| **No prefix (`""`)** | Simpler, cleaner names; recommended when FlexNet is fully retired | Cannot distinguish SLM from FlexNet features during migration; potential name collisions with non-Schrodinger licenses |

> **Recommendation:** If you have fully transitioned to SLM and no longer run FlexNet, use `-prefix ""` for simplicity. The prefix must match the `schrodinger_licensing_namespace` setting in `jobserver.yml` (set it to `""` there too). If you are mid-migration or share a Slurm accounting database with clusters that still use FlexNet, keep the `SLM_` prefix.

---

## Local License Checking: slurm.conf Setup

**Step 3: Add to slurm.conf**

```ini
# slurm.conf
Licenses=MMLIBS:100,GLIDE_SP:50,GLIDE_XP:25,DESMOND_GPGPU:16,FEP_GPGPU:8

# Required for license-aware job requeuing
JobRequeue=1
RequeueExit=99
```

Restart slurmctld after adding:

```bash
systemctl restart slurmctld
```

**Step 4: Verify**

```bash
scontrol show lic
# LicenseName=MMLIBS
#     Total=100 Used=0 Free=100 Remote=no
# LicenseName=GLIDE_SP
#     Total=50 Used=0 Free=50 Remote=no
```

> **Note:** If your license entitlements change (new features added, counts updated), you must re-run `licutil -slurmconf`, update `slurm.conf`, and restart slurmctld. There is no automatic discovery of license changes in local mode.

---

## Remote License Checking (Mixed Environment)

If Schrodinger licenses are consumed both by Slurm jobs and by external users (workstations, laptops via VPN, other clusters), the static `Licenses=` approach will be inaccurate -- Slurm won't know about external checkouts. In this case, use the **license sensor** to dynamically track actual license availability.

**How it works:** The sensor script (`slurm-license-sensor`) polls the SLM license server every 30 seconds, compares current checkouts against the total, and updates the Slurm accounting database. The `CLUSTER_DNS_NAME_REGEX` setting tells the sensor which checkouts are from Slurm nodes (tracked by Slurm) vs. external machines (which reduce the available count).

**Step 1: Configure the sensor**

```bash
# Edit $SCHRODINGER/utilities/slurm-license-sensor
CLUSTER_DNS_NAME_REGEX="compute-.*\\.cluster\\.local"   # Match your compute node hostnames
SLURM_CLUSTERS=["mycluster"]                            # Your Slurm cluster name(s)
SLURM_INSTALLDIR="/opt/slurm"
SCHRODINGER_LICENSING_NAMESPACE="SLM_"                   # Match your prefix choice (or "")
```

---

## Remote License Sensor: systemd Setup

**Step 2: Create a systemd timer**

```ini
# /etc/systemd/system/slurm-license-sensor.service
[Unit]
Description=Schrodinger license sensor for Slurm
After=slurmctld.service slurmdbd.service

[Service]
Type=simple
User=slurm
ExecStart=/opt/schrodinger2026-1/run /opt/schrodinger2026-1/utilities/slurm-license-sensor

[Install]
WantedBy=multi-user.target
```

```ini
# /etc/systemd/system/slurm-license-sensor.timer
[Unit]
Description=Schrodinger license sensor for Slurm

[Timer]
OnCalendar=*:*:0/30
AccuracySec=1s

[Install]
WantedBy=timers.target
```

```bash
chmod +x $SCHRODINGER/utilities/slurm-license-sensor
systemctl enable --now slurm-license-sensor.timer
```

---

## Remote License Sensor: Verification

The sensor automatically registers license resources in the Slurm accounting database:

```bash
# The sensor creates entries like:
sacctmgr -i add resource name=MMLIBS count=100 \
    server=schrodinger_mycluster cluster=mycluster \
    servertype=flexlm percentallowed=100
```

For multi-cluster environments, each cluster gets its own server namespace with `percentallowed=100`, avoiding the need to pre-allocate fixed percentages.

**Step 3: Verify**

```bash
scontrol show lic | grep -A1 mmlibs
# LicenseName=MMLIBS
#     Total=100 Used=0 Free=100 Remote=yes
```

Note `Remote=yes` -- this indicates the license count is managed by the accounting database (sensor-driven) rather than a static slurm.conf entry.

---

## Caveats and Limitations

- **Update latency:** The sensor runs every 30 seconds. Brief race conditions are possible where a job starts before an external checkout is registered.
- **Suite licenses (PACKAGE):** If features A and B are both part of a license package, checking out A does not automatically update B's count. Suite licenses are not perfectly tracked.
- **Multi-server aggregation:** The sensor sums availability across multiple license servers, but a single checkout cannot span servers.
- **Local mode is static:** If using `Licenses=` in slurm.conf (no sensor), license counts do not update when entitlements change -- you must manually re-run `licutil` and restart slurmctld.

---

## Testing License Integration

```bash
# Check license status in Slurm
scontrol show lic | grep -A1 mmlibs

# Submit a test job that checks out licenses
$SCHRODINGER/testapp -HOST cpu -t 300 -l MMLIBS:4 -j testapp.mmlibs_4

# Verify license count updated while job runs
scontrol show lic | grep -A1 mmlibs
# Total=<N> Used=4 Free=<N-4> Remote=yes
```

## SCHRODINGER_LICENSE_RETRY

Set this environment variable in host entries to handle transient license server issues:

```
# In schrodinger.hosts
env: SCHRODINGER_LICENSE_RETRY=5

# In hosts.yml
env:
  - SCHRODINGER_LICENSE_RETRY=5
```

This retries license checkout up to 5 times before failing. Useful for SLM Cloud where network latency may cause occasional transient failures.

---

## Job Server Setup

### Installation

```bash
# Run the setup utility
$SCHRODINGER/utilities/setup_jobserver -setup_jobserver
```

### Configure jobserverd for Slurm

Ensure Slurm binaries are in the PATH for the jobserver user:

```bash
# /etc/profile.d/slurm.sh
export PATH=$PATH:/opt/slurm/bin/:/opt/slurm/sbin/
```

---

## Job Server: Start and Verify

```bash
# Test manually
sudo -u jobserver /bin/bash -c \
    "source /etc/profile.d/slurm.sh && <jobserver_dir>/bin/jobserverd --debug --dir <jobserver_dir>"

# Create systemd service
cat > /etc/systemd/system/jobserverd.service <<EOF
[Unit]
Description=Schrodinger Job Server
After=network.target

[Service]
Type=simple
User=jobserver
ExecStart=/bin/bash -c "source /etc/profile.d/slurm.sh && <jobserver_dir>/bin/jobserverd --dir <jobserver_dir>"
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl enable --now jobserverd
```

### Job Server Ports

| Port | Purpose |
|------|---------|
| 8029 | File server (job data transfer) |
| 8030 | Job server API |

Ensure these ports are accessible from all compute nodes and user workstations.

---

## Testing & Validation

### installation_check

Run Schrodinger's built-in validation:

```bash
$SCHRODINGER/installation_check
```

This tests license connectivity, library loading, and basic functionality.

### Test Queue Submission

```bash
# Submit a test job to each host entry
$SCHRODINGER/testapp -HOST cpu -t 60 -j test_cpu
$SCHRODINGER/testapp -HOST gpu -t 60 -j test_gpu

# Monitor with Job Control
$SCHRODINGER/jobcontrol -list

# Or with Job Server
$SCHRODINGER/jsc list
```

### Verify Slurm Integration

```bash
# Check that the Schrodinger job appears in Slurm
squeue -u $USER

# After completion, check accounting
sacct -j <JOBID> --format=JobID,JobName,Elapsed,MaxRSS,ExitCode
```

---

## Deployment Callouts

> **ParallelCluster Note:** On ParallelCluster, the Schrodinger installation should be on shared storage (FSx Lustre or EFS) accessible from all compute nodes. The `tmpdir` should point to local NVMe instance storage or a fast shared filesystem. Partition names follow the ParallelCluster naming convention (e.g., `cpu-st-c5-4xl`), so update Qargs accordingly. License server access requires VPC security group rules allowing outbound TCP to the license server.

> **PCS Note:** On PCS, install Schrodinger in a custom AMI or on shared storage (EFS/FSx). Configure host entries in your custom AMI's Schrodinger installation. PCS queue names map to your compute node group names.

> **On-Prem Note:** On-prem clusters have the most flexibility. You can mount the Schrodinger installation via NFS, use local scratch for tmpdir, and configure license servers on the local network. For multi-cluster environments with separate Job Servers, ensure host entry names are unique across clusters.

Exercises¶

Map a jsc id to its Slurm job

Submit a short test job through a Job Server host entry, then find the Slurm job that carries its jsc id. Do it from both ends: start at the jsc id and find the Slurm job, then start at the Slurm job and recover the jsc id. Confirm that a direct sbatch job has no such comment.

Hint / Solution

# 1. Submit through a Job Server host entry; note the job id jsc reports
$SCHRODINGER/testapp -HOST cpu -t 60 -j test_cpu
$SCHRODINGER/jsc list | head

# 2. Slurm job -> jsc id (Comment holds SchrodingerJobId=<uuid>)
squeue -u $USER -o '%i %j %k'
scontrol show job <slurm_jobid> | grep -oE 'SchrodingerJobId=[^ ,]*'

# 3. jsc id -> Slurm job: search the comments
squeue -u $USER -h -o '%i %k' | grep <uuid>

# 4. Control: a direct sbatch job has no SchrodingerJobId
sbatch --wrap='sleep 60'
scontrol show job <direct_jobid> | grep -c SchrodingerJobId    # 0

Schrodinger User Guide -- end-user perspective on Schrodinger + Slurm
Resource Management -- GRES and license configuration in Slurm
On-Premise Deployment -- cluster setup for Schrodinger workloads
AWS ParallelCluster -- cloud deployment considerations

Schrodinger Admin

Exercises¶

Related Modules¶

References¶