Environment Modules # Environment Modules ## The Problem HPC clusters host many software packages, often with multiple versions. Without a management system, you'd have conflicting PATH entries, incompatible library versions, and chaos. --- ## What Are Environment Modules? Environment modules dynamically modify your shell environment -- setting `PATH`, `LD_LIBRARY_PATH`, and other variables -- so the right version of software is available when you need it. Two implementations are common: - **Environment Modules** (TCL-based, the original) - **Lmod** (Lua-based, more features, increasingly standard) The commands are nearly identical for daily use. --- ## Essential Commands ```bash # List available modules $ module avail # Search for a specific package $ module avail blast --- /opt/modules --- blast/2.13 blast/2.14 blast/2.15 (D) # Load a module $ module load blast/2.15 # Load default version (marked with D) $ module load blast # List currently loaded modules $ module list Currently Loaded Modules: 1) blast/2.15 # Show what a module does $ module show blast/2.15 # Unload a module $ module unload blast # Swap versions $ module swap blast/2.15 blast/2.14 # Unload everything $ module purge # Search by keyword (Lmod) $ module spider python ``` --- ## Using Modules in Job Scripts Always load your modules in the job script. Don't rely on your login environment being inherited: ```bash #!/bin/bash #SBATCH --job-name=blast_job #SBATCH --time=02:00:00 # Clean environment first (recommended) module purge # Load required software module load blast/2.15 module load python/3.11 # Verify which blastn python --version # Run your work blastn -query input.fasta -db /data/nt -out results.txt ``` > **Why `module purge` first?** It ensures a clean, reproducible environment regardless of what was loaded in your shell before submission. --- ## Module Hierarchies (Lmod) Lmod supports hierarchical modules where loading a compiler or MPI implementation reveals additional modules: ```bash $ module avail # Shows core modules only $ module load gcc/12.3 # Load a compiler $ module avail # Now shows modules built with gcc/12.3 $ module load openmpi/4.1 $ module avail # Now shows MPI-dependent modules too ``` This prevents loading incompatible combinations (e.g., a library built with GCC used with Intel-compiled code). --- ## Module Collections (Lmod) Save a set of modules for reuse: ```bash # Set up your environment $ module purge $ module load gcc/12.3 openmpi/4.1 python/3.11 blast/2.15 # Save as a named collection $ module save myenv # Later, restore it $ module restore myenv # List saved collections $ module savelist ``` > **ParallelCluster / PCS Note:** Because compute nodes are launched dynamically, software modules must be installed on **shared storage** (EFS or FSx for Lustre) so they are available on every node. A common pattern is to mount a shared filesystem at `/opt/modules` or `/shared/modules` and point `MODULEPATH` there. Modules installed only on the head node's local disk will not be visible to compute nodes. --- ## Common Pitfalls **Module not found in job script:** Your login node and compute nodes must have the same module paths. If a module works interactively but fails in a job, check that the module system is initialized: ```bash #!/bin/bash #SBATCH ... # This is usually automatic, but if modules fail: source /etc/profile.d/modules.sh # or lmod.sh module load blast/2.15 ``` **Wrong version loaded:** Be explicit about versions. `module load blast` loads the default, which may change when admins update it. **Conflicting modules:** Lmod will warn or error if you try to load incompatible modules. Use `module swap` to switch versions. References¶ Lmod Documentation Environment Modules Project