Modules and bcmodule

The Baseline Configuration Team provides bcmodule, a command which executes like the standard module command but has numerous improvements and new features. Users can think of bcmodule as a total replacement for module, though technically it is a "wrapper" which calls module. The two commands can be used interchangeably in the same shell, except when one of the new features of bcmodule is needed.

First we give an explanation of modules in general and then we discuss bcmodule in particular.

What are modules and what do they do?

The purpose of the module command is to establish the proper environment in the user's shell for running an application. The shell itself contains information in two types of variables that are called "shell variables" and "environment variables." The information in these variables is used to locate executables, data files and shared libraries. Having the shell environment variables set correctly can make the difference between being able to run an application and not.

A module command for an application typically augments the user's PATH variable so that the executable for that application can be located. It may also set an environment variable, such as LD_LIBRARY_PATH, which tells the operating system where to look for libraries the executable requires to run. It may set other environment variables that contain the location of required data files. Without modules, users would need to know where an application resides, how to add that path to the PATH variable, which libraries are needed, where they are, how to tell the operating system how it can find them, etc. So using modules makes computing much easier, even for experts.

How does one use modules?

The basics one needs to know about modules include the module command, the module sub-commands, module directories, and modulefiles.

Let's start with the modulefiles. Each version of an application, compiler, MPI, etc. will have its own modulefile, which is a file containing the information required to initialize the shell environment to run that version of the application, compiler, or MPI. To configure a shell, a user executes a module load on the modulefile. This means that the module command reads the modulefile, and modifies the shell accordingly. Strictly speaking, module is the command, but instead of speaking of "loading a modulefile", one usually speaks of "loading a module".

Before a user can load a module, the module command has to know where the modulefile is. To find a modulefile, a user's shell must be aware of the directory that the modulefile resides in. The module command finds a modulefile using the same method the shell uses to find a command: it searches along a path, which is an ordered list of directories, until it finds the file it is looking for. This path is stored in the shell as the environment variable MODULEPATH. You can see its value by executing echo $MODULEPATH. To set this path to include directory_1, one executes module use directory_1. In executing this, the module command reads the current value of $MODULEPATH and adds directory_1 to it, thus modifying your shell, which is now aware of one more directory where modulefiles can be found.

Now that the module command can find the modulefile, we are ready to have the modulefile modify a shell (i.e. to have it "work"). This is done by executing a module load, such as module load compiler/gcc/4.9.3. The load in this command is called the sub-command, and the modulefile is compiler/gcc/4.9.3. The portion of the full path to the modulefile that precedes the modulefile name is the directory that was added to MODULEPATH earlier by module use. To remove a directory from your MODULEPATH, execute module unuse directory_2. This will make the modulefiles in directory_2 disappear from your shell's point of view.

Module commands always have the form module switches sub-command args. To say a module is loaded means that it is in effect (i.e. your shell is configured according to that modulefile). If a user wishes to run a different version of the same application, then he/she must unload the current module, and then execute a module load on the desired version. A module unload removes the additions to the shell environment that the corresponding module load had made. Executing module swap can be used to unload one module and load another in one command. A module can load one or more other modules.

Users typically have multiple modules loaded at any one time (just not multiple versions of a particular application). To see which modules are currently loaded, execute module list. To see which modules one can load, execute module avail. Executing module use directory as described above will make more modules available to your shell. To see what a module does to your shell, execute module show . To see a list of all module sub-commands execute module with no arguments, or module --help (notice there are two dashes).

A summary of the more important module subcommands is given below:

Module commands that modify the shell:
Command Details
module use mod_directory add a directory to $MODULEPATH
module unuse mod_directory remove a directory from $MODULEPATH
module load modulefile modifies the shell to run an application
module unload modulefile undoes what the module load did
module swap mod_a mod_b unload module mod_a and load mod_b
Module commands that provide information
without modifying the shell
Command Details
module show module_name show what the module does to your shell
module list list which modules are currently loaded
module avail list which modules could be loaded

The staff at the DSRCs maintains modulefiles, some of which come from vendors, and some are generated locally. Users who run their own codes can even write their own modulefile if they wish. To do so, you may copy and then amend an existing modulefile, or ask for help from the HPC Help Desk.

The BCT command "bcmodule"

The command bcmodule works in all six shells supported by the HPCMP: sh, bash, ksh, zsh, csh, and tcsh. Both module and bcmodule should be initialized by default upon login. Execute bcmodule to see the help menu of sub-commands.

Since bcmodule can be thought of as a replacement for module, it can be used to list the modules that are currently loaded. Below illustrates an example, using the C-shell (csh or tcsh). Notice that sub-command list has been abbreviated to li; such abbreviations are optional.

$ bcmodule li
Currently Loaded Modulefiles:
 1) compiler/intel/2017.1.132   2) mpi/sgimpt/2.15   3) Master>

So three modules are loaded. To see what a particular module does to your shell, execute the show sub-command as shown below.

$ bcmodule show Master
-------------------------------------------------------------------
/p/app/modulefiles/unsupported/Master:

setenv           BC_HOST systemname
setenv           BC_CORES_PER_NODE 40
setenv           BC_STANDARD_NODE_CORES 40
setenv           BC_ACCELERATOR_NODE_CORES 40
setenv           BC_BIGMEM_NODE_CORES 40
setenv           BC_PHI_NODE_CORES 0
setenv           BC_NODE_TYPE LOGIN
setenv           MODULE_VERSION 3.2.10
setenv           CSI_HOME /usr/cta
setenv           DAAC_HOME /usr/cta/DAAC
setenv           PET_HOME /usr/cta/pet
setenv           WORKDIR /p/work1/username
setenv           SAMPLES_HOME /usr/cta/SCR
setenv           CENTER /p/cwfs/username
setenv           MPSCP_CONFIG_FILENAME /etc/mpscp_config
setenv           MPSCP_BLOCKED_FILE /etc/mpscp_blocked_ports
setenv           PROJECTS_HOME /usr/cta/unsupported
setenv           KRB5HOME /usr/krb5/bin
setenv           KRB5_HOME /usr/krb5/bin
setenv           COST_HOME /p/app/unsupported/COST
setenv           JAVA_HOME /opt/rh/rh-eclipse46/root/usr/lib/jvm/java
setenv           ARCHIVE_HOME /admin/username
setenv           ARCHIVE_HOST archive.domain
prepend-path     MANPATH /usr/share/man:/usr/local/man
prepend-path     PATH /usr/local/bin:/usr/krb5/bin
prepend-path     PATH //usr/cta/unsupported/BC
prepend-path     PATH /usr/people/PBS/SLB
module           load compiler/intel/2017.1.132 mpi/sgimpt/2.15
-------------------------------------------------------------------

The first line (starting with /usr/cta) shows where the Master modulefile is located. This is followed by 23 setenv lines, each of which sets the value of an environment variable. When you run a Linux command or execute an application, it can extract the value of an environment variable defined in the shell. For example, when I execute the command archive to archive a file, the archive command knows where to copy the file based on my shell's value of the variable ARCHIVE_HOME, which is my personal directory /admin/username. In this case, the Master module sets a unique value for ARCHIVE_HOME for each user, so that one user's files are not archived into another user's directory.

The last four lines of the module modify an existing shell variable. The first prepend-path module statement prepends two directories (/usr/share/man and /usr/local/man) to the variable MANPATH, which is used in finding man pages (Linux online documentation). The last three lines each insert one or two directories at the beginning of the variable PATH, which is used to find Linux commands as well as application scripts and executables.

Here is a list of improvements provided by bcmodule that are not available in module:

  • It translates native modulefile names into BCT-specified modulefile names for greater uniformity of modulefile names across different systems in the HPCMP. Users can use either the BCT modulefile name or the native modulefile name interchangeably. Two new sub-commands (tran and show_all_trans) show the translation of modulefile names from native to BCT and vice-versa, as illustrated below. 
    $ bcmodule tran compiler/gcc/6.2.0
real module name is gcc/6.2.0
BCT  module name is compiler/gcc/6.2.0
  • It returns non-zero return codes when errors occur.
  • Output from bcmodule goes to stdout, which allows users to pipe the output into subsequent Linux commands such as grep or more, as illustrated below. 
    $ bcmodule avail | grep perftools

cray-petsc-complex-64/3.6.3.0       perftools-base/6.3.0
cray-petsc-complex-64/3.7.4.0       perftools-base/6.3.2
cray-petsc/3.6.3.0                  perftools/6.4.4
    This does not work with module since it writes its output to standard error. Each line of output from the bcmodule command above has perftools in it, but it also contains modules that have nothing to do with perftools. This is because grep is grabbing entire lines. Next we will show a better bcmodule feature for seeing only the perftools related modules.
  • A new sub-command find searches through all known modulefile directories (not just those in use) and lists all modulefile names it finds that contain the given string searched for, thus helping users find modulefiles when the exact name is uncertain and/or the location is unknown. An illustration of the sub-command find is included below. 
    $ bcmodule find perftools
/opt/cray/modulefiles/perftools-lite
total 36
-rwxr-xr-x 1 root root 7794 Jan 15  2015 6.2.2
-rwxr-xr-x 1 root root 7201 Aug 25  2015 6.2.5
-rw-r--r-- 1 root root 7616 Oct 19  2015 6.3.0
-rw-r--r-- 1 root root 7817 Apr 13  2016 6.3.2
-rw-r--r-- 1 root root 1506 Jan 25  2017 6.4.4

/opt/cray/modulefiles/perftools-base
total 28
-rw-r--r-- 1 root root 7707 Oct 19  2015 6.3.0
-rw-r--r-- 1 root root 8015 Apr 13  2016 6.3.2
-rw-r--r-- 1 root root 9707 Jan 25  2017 6.4.4

/opt/cray/modulefiles/perftools
total 36
-rwxr-xr-x 1 root root 7794 Jan 15  2015 6.2.2
-rwxr-xr-x 1 root root 7201 Aug 25  2015 6.2.5
-rw-r--r-- 1 root root 7560 Oct 19  2015 6.3.0
-rw-r--r-- 1 root root 7761 Apr 13  2016 6.3.2
-rw-r--r-- 1 root root 1506 Jan 25  2017 6.4.4
    Here the output format is a bit different, but all the information you need to locate the perftools modules is given. And the timestamps of the files is also given, which is not the case for module commands.
  • A new sub-command find_in_module searches for a modulefile which contains the given string, thus locating the modulefile not by a string in its name (the sub-command find described above does that), but by finding a modulefile that performs any action involving the string. For example, bcmodule find_in_module XYZ finds any modulefile that sets the environment variable XYZ, or that sets any environment variable with XYZ as part of the name, or augments any PATH with a directory having the pattern XYZ, etc.

    An illustration of the subcommand find_in_module is included below. Each modulefile listed has the string ARCH somewhere in its contents.

    $ bcmodule find_in_module ARCH
/p/app/modulefiles/unsupported/master
/p/app/modulefiles/unsupported/.Master.old
/p/app/modulefiles/unsupported/Master
/p/app/modulefiles/apps/fluent/172~
/p/app/modulefiles/apps/fluent/180
/p/app/modulefiles/apps/fluent/182
  • A new sub-command inuse lists all module directories currently in use.
  • A new sub-command show_all_dirs lists all known module directories so that users can easily add module directories for greater access to modulefiles.
  • Even easier, the new sub-command use_all_dirs adds all known module directories so that users automatically use all configured module directories when loading and listing available modules.


    • Below demonstrates how the number of modules available increases (from 41 lines of output to 100 lines) when all (system configured) modulefile directories are added using the use_all_dirs option.

    $ bcmodule avail | wc -l
41
$ bcmodule use_all_dirs
$ bcmodule avail | wc -l
100
    These three commands show that by executing bcmodule use_all_dirs, the list of module directories known to the shell increases such that an additional 59 lines worth of modules are now locatable, and can thus be loaded. The inuse and show_all_dirs sub-commands mentioned above can be used to see those directories.

    The term "system configured" used above refers to the commonly used module directories configured for bcmodule by the system admin, but not directories containing only modules for certain projects. Users who wish to always have their project specific module directory used can do so by including a module use command in their personal shell initialization script.
  • A new sub-command count counts modulefiles and modulefile directories currently used.
  • A new switch (-set_gcc) loads the gcc module when a compiler module is loaded (this facilitates compiling under certain circumstances).
  • A new switch (-set_gxx) sets the environment variable GXX_ROOT when a compiler module is loaded (this facilitates compiling under certain circumstances).

    • Tired of getting too much output from module avail? The new bcmodule features find, -compress, -sf, -xf, -sd, and -xd all make finding modulefiles much easier.

  • A new option for module avail: -compress removes the module version and lists each "family" of modules once instead of once per version. On one system, this reduced the number of lines of output from 202 to 51.

    Below includes an example where module avail, for the directory /opt/modulefiles, gives 21 lines of output whereas the -compress option in bcmodule shortens the output to just two lines:

    $ module avail

--------------------------------------- /opt/modulefiles ----------------------------------------
cce/8.3.14(default)                   intel/14.0.2.144
cce/8.3.5                             intel/15.0.1.133
cce/8.3.6                             intel/16.0.0.109
cce/8.3.9                             intel/17.0.1.132
cce/8.4.0                             java/jdk1.7.0_45
cce/8.4.1                             java/jdk1.8.0_51(default)
cce/8.4.5                             modules/3.2.10.3(default)
cce/8.5.6                             modules/3.2.10.5
chapel/1.9.0.1(default)               modules/3.2.6.7
ddt/4.2.2.6_39982(default)            pbs
ddt/5.0.1.3_42607                     pgi/14.10.0
ddt-memdebug/4.2.2.6_39982(default)   pgi/14.10.0-acc
ddt-memdebug/5.0.1.3_42607            pgi/15.3.0(default)
eswrap/1.3.3-1.020200.1278.0(default) pgi/15.3.0-acc
gcc/4.8.1                             pgi/15.7.0
gcc/4.9.2                             pgi/15.7.0-acc
gcc/4.9.3(default)                    pgi/16.10.0
gcc/5.1.0                             pgi/16.10.0-acc
gcc/5.3.0                             xc-sysroot/5.2.40
gcc/6.1.0                             xc-sysroot/5.2.82(default)
gcc/6.2.0

$ bcmodule avail -compress

--------------------------------------- /opt/modulefiles ---------------------------------------
cce     compiler/gcc    compiler/java  ddt  ddt-memdebug  eswrap  modules  pbs  xc-sysroot
chapel  compiler/intel  compiler/pgi
  • New options for module avail and module list that reduce the amount of output:
    a. -sf pattern lists only modulefiles having pattern in the name.
    b. -xf pattern lists only modulefiles not having pattern in the modulefile name.

    An illustration of the sub-command av with option -sf is below.

    $ bcmodule av -sf perftool

------------------------------------ /opt/cray/modulefiles -------------------------------------
perftools-base/6.3.0  perftools-lite/6.2.5(default)  perftools-lite/6.4.4      perftools/6.3.0
perftools-base/6.3.2  perftools-lite/6.3.0           perftools/6.2.2           perftools/6.3.2
perftools-base/6.4.4  perftools-lite/6.3.2           perftools/6.2.5(default)  perftools/6.4.4
perftools-lite/6.2.2
  • New options for module avail:
    a. -sd dir lists only modulefiles in directories having dir as part of the directory name.
    b. -xd dir lists only modulefiles in directories not having dir as part of the directory name.


    The two commands bcmodule find pattern and module avail –sf pattern are very similar, but not identical. Their output formats are different, but more importantly, find searches through all configured modulefile directories, whereas the second command only searches through directories you have in use.

    • The new bcmodule does a slightly better job than module of "cramming" as many columns of output as it can to your screen to reduce the number of lines of output. In the process of reformatting the output, it sometimes alphabetizes the modulefiles slightly differently.

    Happy moduling!