HPC2020: The Lmod Module system

Lmod is a Lua-based implementation of the module system, and generally claimed to be backwards compatible with the original. On the newer ECMWF platforms it is replacing the traditional TCL modules.

It is used to configure and manage your session or job environment with all those tools and libraries required for your workload.

Table of contents

Main features

• All the shells in use at the centre are supported.
• Same command and almost identical syntax and options as TCL modules.
• Graceful failure in case of error. TCL modules sometimes fails and gives a zero exit code which cannot be trapped. Some users have implemented their own functions to workaround this issue in the past.
• It does the right thing: automatic swap if different version of module is loaded. With TCL modules you need to know if that module is loaded to decide whether to swap or to load. Many times you end up unloading and loading to make it easier.
• Subcommands like avail or list work on STDOUT so piping the output to other commands such as grep is much more convenient.
• You can define your own module collections and load them as a whole.
• The ml shortcut.
• Avail mode also shows those modules which are loaded.
• Better support for dependencies and conflicts to other modules.
• Native support to manage different toolchains (different compiler families and versions, and even MPI flavours).

Behaviour change: Lmod does the right thing

If you have used modules before, you will see the commands and options are almost identical. You will be able to load and unload modules, as well as checking which ones are loaded or available to load.

Please have a look at the Lmod modules command reference list for details on the different subcommands.

However, there are some changes in the behaviour in respect to the Traditional TCL modules that you may appreciate:

Return codes

If the command fails for any reason (i.e. the module does not exist), it will return a non-zero exit code so you can trap it and deal with it appropriately. There is no need to check wether the module has been loaded or not. You may find that some scripts may fail early where they were not failing before because of this new behaviour

$ module load git; echo return_code=$?
return_code=0
$ module load foo; echo return_code=$?
Lmod has detected the following error:  The following module(s) are unknown: "foo"

Please check the spelling or version number. Also try "module spider ..."
It is also possible your cache file is out-of-date; it may help to try:
  $ module --ignore-cache load "foo"

Also make sure that all modulefiles written in TCL start with the string #%Module

return_code=1

Load and swap

Loading a module when there was another version of it loaded is no longer a problem. Lmod will swap them automatically.

$ module load git
$ module load git/new

The following have been reloaded with a version change:
  1) git/2.20.1 => git/2.25.1

Loaded modules

You can see how many modules and which versions are loaded with "module list"

$ module list

Currently Loaded Modules:
  1) gcc/8.5.0   2) prgenv/gnu   3) git/2.30.1

Automatic version aliases

If you try to load a module with an incomplete version, it will try and find the best (latest) version that matches instead of failing.

$ module avail git

-------------------------------------------------------------------------------------- Global Aliases ---------------------------------------------------------------------------------------
   pa -> prgenv/amd    pe -> prgenv/expert    pg -> prgenv/gnu    pi -> prgenv/intel    pp -> prgenv/pgi

--------------------------------------------------------------------------- /usr/local/apps/modulefiles/lmod/core ---------------------------------------------------------------------------
   git/2.20.1 (D)    git/2.25.1 (new)

  Where:
   Aliases:  Aliases exist: foo/1.2.3 (1.2) means that "module load foo/1.2" will load foo/1.2.3
   D:        Default Module

Use "module spider" to find all possible modules and extensions.
Use "module keyword key1 key2 ..." to search for all possible modules matching any of the "keys".

$ module load git/2
$ module load git/2.25

The following have been reloaded with a version change:
  1) git/2.20.1 => git/2.25.1

Conflict management

If two modules cannot be loaded at the same time due to a conflict, the old one will be swapped out and the new one will be loaded automatically.

$ module load proj4
$ module load proj

Lmod is automatically replacing "proj4/5.2.0" with "proj/6.1.1".

Native flavour support

You may feel that some packages or versions are missing when running the classic module avail. Lmod knows about those packages that are flavoured for different environments (compilers or MPI), and will only show as available to load those that have been installed for the currently loaded environment. This guarantees that when loading a module the right flavour of the tool or library will be used, avoiding incompatibilities.

Use module spider to see the whole picture.

The prgenv module

Different compilers families and versions are provided, and with them many different flavours of a given package and versions may be available. Loading the right one matching the current setup avoiding incompatibilities with other packages becomes a complex task.

The prgenv module addresses this problem. There is a prgenv for each compiler family, so it can guarantee a minimal compatibility between different packages.

$ module avail prgenv

-------------------------------------------------------------------------------------- Global Aliases ---------------------------------------------------------------------------------------
   pa -> prgenv/amd    pe -> prgenv/expert    pg -> prgenv/gnu    pi -> prgenv/intel    pp -> prgenv/pgi

------------------------------------------------------------------------- /usr/local/apps/modulefiles/lmod/prgenvs --------------------------------------------------------------------------
   prgenv/amd (a)    prgenv/expert (E,e)    prgenv/gnu (D:g)    prgenv/intel (i)    prgenv/pgi

The ECMWF setup: things to remember

Here are a few considerations regarding the setup at ECMWF:

• No modules are loaded by default, beyond the system gcc compiler and associated prgenv. You should load what you need when you need it. If you really want to have some default modules, you may add the corresponding module loads into your ~/.profile or use a collection.
• You may need to load a prgenv, and optionally an MPI implementation, to be able to see a more complete list of modules to load.
• Use module spider if you can't find a module. If it exists, it will tell you how to load it.