Environment Modules

Overview

Teaching: 35 min
Exercises: 5 min

Topics

• How to load modules to access software that I want to use for my research?

Objectives

• Learn about modules, how to search, load and unload modules

Environment Variables

The shell, and many other command line programs uses a set of variables to control their behavior. Those variables are called Environment Variables. Think about them as placeholders for information stored within the system that passes data to programs launched in the shell.

Environment Variables control CLI functionality. They declare where to search for executable commands, where to search for libraries, which language display messages to you, how you prompt looks. Beyond the shell itself, Environment Variables are use by many codes to control their own operation.

You can see all the variables currently defined by executing:

$ env

Shell variables can be created like

$ A=10
$ B=20

Environment variables are shell variables that are exported, ie converted into global variables, the command to do this could be like:

$ A=10
$ B=20
$ export A
$ export B

Or simply:

$ export A=10
$ export B=20

Environment variables are similar to the shell variables that you can create of the shell. Shell variables can be used to store data and manipulated during the life of the shell session. However, only environment variables are visible by child processes created from that shell. To clarify this consider this script:

#!/bin/bash

echo A= $A
echo B= $B
C=$(( $A + $B ))
echo C= $C

Now create two shell variables and execute the script, do the same with environment variables and notice that now the script is able to see the variables.

Some common environment variables are:

Those are just a few environment variables of common use. There are many more. Changing them will change where executables are found, which libraries are used and how the system behaves in general. That is why managing the environment variables properly is so important on a machine, and even more on a HPC cluster, a machine that runs many different codes with different versions.

Here is where environment modules enters.

Environment Modules

The modules software package allows you to dynamically modify your user environment by using modulefiles.

Each modulefile contains the information needed to configure the shell for an application. After the modules software package is initialized, the environment can be modified on a per-module basis using the module command, which interprets modulefiles. Typically, modulefiles instruct the module command to alter or set shell environment variables such as PATH, MANPATH, and others. The modulefiles can be shared by many users on a system, and users can have their own collection to supplement or replace the shared modulefiles.

As a user, you can add and remove modulefiles from the current environment. The environment changes contained in a modulefile can also be summarized through the module show command. You are welcome to change modules in your .bashrc or .cshrc, but be aware that some modules print information (to standard error) when loaded, this should be directed to a file or /dev/null when loaded in an initialization script.

Basic arguments

The following table lists the most common module command options

Creating a private repository

The basic procedure is to locate modules on a folder accessible by relevant users and add the variable MODULEPATH to your .bashrc

MODULEPATH controls the path that the module command searches when looking for modulefiles. Typically, it is set to a default value by the bootstrap procedure. MODULEPATH can be set using ’module use’ or by the module initialization script to search group or personal modulefile directories before or after the master modulefile directory.

Exercise: Using modulefiles

1. Check the modules that you currently have and clean (purge) your environment from them. Check again and confirm that no module is loaded.

2. Check which versions of Python, R and GCC you have from the RHEL itself. Try to get and idea of how old those three components are. For python and R all that you have to do is enter the corresponding command (R or python). For GCC you need to use gcc --version and see the date of those programs.

3. Now lets get newer version of those 3 components by loading the corresponding modules. Search for the module for Python 3.7.2 and R 3.4.1 and GCC 8.2.0 and load the corresponding modules. To make things easier, you can use check the availability of modules just in the languages section.

   module avail lang
   

4. Check again which version of those 3 components you have now. Notice that in the case of Python 3, the command python still goes towards the old python 2.6.6, as the python 3.x interpreter is not backwards compatible with python 2.x the new command is called python3, check its version by entering the command.

5. Clean all of the environment

   module purge
   

6. Go back and purge all the modules from your environment. We will now explore why it is important to use a recent compiler. Try to compile the code at workshops_hands-on/Introduction_HPC/5._Environment_Modules/lambda_c++14.cpp. Go to the folder and execute:

   g++ lambda_c++14.cpp
   

   At this point you should have received a list of errors, that is because even if the code is C++ it uses elements of the language that were not present at that time on C++ Specification. The code actually uses C++14 and only recent versions of GCC allows for these declarations. Lets check how many GCC compilers we have available on Thorny Flat.

   module avail lang/gcc
   

   Now from that list, start loading and trying to compile the code as indicated above. Which versions of GCC allow you to compile the code? Also try the Intel compilers. In the case of intel the command to compile the code is

   icpc lambda_c++14.cpp
   

   Try with all the Intel compilers, it will fail with all of them. That is because the default standard for the Intel C++ compiler is not C++14, you need to declare it explicitly and only for Intel Compiler suite 17.0.1

   icpc lambda_c++14.cpp -std=c++14
   

   Now it should be clearer why modules is an important feature of any HPC infrastructure as it allows you to use several compilers, libraries and packages in different versions. On a normal computer, you usually have just one.

Reference: Modules on the clusters

This is the list of all modules on Thorny Flat on July 2020.

Key Points

• Use module avail to know all the modules on the cluster.

• Use module load <module_name> to load the module that you need.

• You can preload modules for each login by adding the load line on your $HOME/.bashrc