Modules and bcmodule
The Baseline Configuration Team provides
bcmodule, a command which executes like the standard
module command but has numerous improvements and new features. Users can think of
as a total replacement for module, though technically it is a "wrapper" which calls
module. The two
commands can be used interchangeably in the same shell, except when one of the new features of
First we give an explanation of modules in general and then we discuss
bcmodule in particular.
What are modules and what do they do?
The purpose of the module command is to establish the proper environment in the user's shell for running an application. The shell itself contains information in two types of variables that are called "shell variables" and "environment variables." The information in these variables is used to locate executables, data files and shared libraries. Having the shell environment variables set correctly can make the difference between being able to run an application and not.
module command for an application typically augments the user's
PATH variable so that the
executable for that application can be located. It may also set an environment variable, such as
which tells the operating system where to look for libraries the executable requires to run. It may set other environment
variables that contain the location of required data files. Without modules, users would need to know where an
application resides, how to add that path to the
PATH variable, which libraries are needed, where
they are, how to tell the operating system how it can find them, etc. So using modules makes computing much
easier, even for experts.
How does one use modules?
The basics one needs to know about modules include the module command, the module sub-commands, module directories, and modulefiles.
Let's start with the modulefiles. Each version of an application, compiler, MPI, etc. will have its own modulefile,
which is a file containing the information required to initialize the shell environment to run that version of the
application, compiler, or MPI. To configure a shell, a user executes a
module load on the modulefile.
This means that the
module command reads the modulefile, and modifies the shell accordingly.
module is the command, but instead of speaking of "loading a modulefile", one usually
speaks of "loading a module".
Before a user can load a module, the module command has to know where the modulefile is. To find a modulefile, a user's
shell must be aware of the directory that the modulefile resides in. The module command finds a modulefile using the same
method the shell uses to find a command: it searches along a
path, which is an ordered list of directories,
until it finds the file it is looking for. This path is stored in the shell as the environment variable MODULEPATH. You
can see its value by executing
echo $MODULEPATH. To set this path to include
module use directory_1. In executing this, the module command reads the current value of
$MODULEPATH and adds
directory_1 to it, thus modifying your shell, which is now aware of one
more directory where modulefiles can be found.
Now that the module command can find the modulefile, we are ready to have the modulefile modify a shell (i.e. to have
it "work"). This is done by executing a
module load, such as
module load compiler/gcc/4.9.3.
load in this command is called the sub-command, and the modulefile is
The portion of the full path to the modulefile that precedes the modulefile name is the directory that was added to
MODULEPATH earlier by
module use. To remove a directory from your MODULEPATH, execute
directory_2. This will make the modulefiles in
directory_2 disappear from your shell's point of
Module commands always have the form
module switches sub-command args. To say a module is loaded means
that it is in effect (i.e. your shell is configured according to that modulefile). If a user wishes to run a different
version of the same application, then he/she must
unload the current module, and then execute a module
load on the desired version. A module
unload removes the additions to the shell environment that the
corresponding module load had made. Executing
module swap can be used to unload one module and load
another in one command. A module can load one or more other modules.
Users typically have multiple modules loaded at any one time (just not multiple versions of a particular application).
To see which modules are currently loaded, execute
module list. To see which modules one can load, execute
module avail. Executing
module use directory as described above will make more modules
available to your shell. To see what a module does to your shell, execute
module show To see
a list of all
module sub-commands execute module with no arguments, or
module --help (notice
there are two dashes).
A summary of the more important
module subcommands is given below:
|module use mod_directory||add a directory to $MODULEPATH|
|module unuse mod_directory||remove a directory from $MODULEPATH|
|module load modulefile||modifies the shell to run an application|
|module unload modulefile||undoes what the module load did|
|module swap mod_a mod_b||unload module mod_a and load mod_b|
|module show module_name||show what the module does to your shell|
|module list||list which modules are currently loaded|
|module avail||list which modules could be loaded|
The staff at the DSRCs maintains modulefiles, some of which come from vendors, and some are generated locally. Users who run their own codes can even write their own modulefile if they wish. To do so, you may copy and then amend an existing modulefile, or ask for help from the HPC Help Desk.
The BCT command "bcmodule"
bcmodule works in all six shells supported by the HPCMP: sh, bash, ksh, zsh, csh,
and tcsh. Both module and bcmodule should be initialized by default upon login. Execute
to see the help menu of sub-commands.
Since bcmodule can be thought of as a replacement for module, it can be used to list the modules that are
currently loaded. Below illustrates an example, using the C-shell (csh or tcsh). Notice that sub-command
list has been abbreviated to
li; such abbreviations are optional.
$ bcmodule li Currently Loaded Modulefiles: 1) compiler/intel/2017.1.132 2) mpi/sgimpt/2.15 3) Master>
So three modules are loaded. To see what a particular module does to your shell, execute the
sub-command as shown below.
$ bcmodule show Master ------------------------------------------------------------------- /p/app/modulefiles/unsupported/Master: setenv BC_HOST systemname setenv BC_CORES_PER_NODE 40 setenv BC_STANDARD_NODE_CORES 40 setenv BC_ACCELERATOR_NODE_CORES 40 setenv BC_BIGMEM_NODE_CORES 40 setenv BC_PHI_NODE_CORES 0 setenv BC_NODE_TYPE LOGIN setenv MODULE_VERSION 3.2.10 setenv CSI_HOME /usr/cta setenv DAAC_HOME /usr/cta/DAAC setenv PET_HOME /usr/cta/pet setenv WORKDIR /p/work1/username setenv SAMPLES_HOME /usr/cta/SCR setenv CENTER /p/cwfs/username setenv MPSCP_CONFIG_FILENAME /etc/mpscp_config setenv MPSCP_BLOCKED_FILE /etc/mpscp_blocked_ports setenv PROJECTS_HOME /usr/cta/unsupported setenv KRB5HOME /usr/krb5/bin setenv KRB5_HOME /usr/krb5/bin setenv COST_HOME /p/app/unsupported/COST setenv JAVA_HOME /opt/rh/rh-eclipse46/root/usr/lib/jvm/java setenv ARCHIVE_HOME /admin/username setenv ARCHIVE_HOST archive.domain prepend-path MANPATH /usr/share/man:/usr/local/man prepend-path PATH /usr/local/bin:/usr/krb5/bin prepend-path PATH //usr/cta/unsupported/BC prepend-path PATH /usr/people/PBS/SLB module load compiler/intel/2017.1.132 mpi/sgimpt/2.15 -------------------------------------------------------------------
The first line (starting with /usr/cta) shows where the
Master modulefile is located. This
is followed by 23
setenv lines, each of which sets the value of an environment variable. When
you run a Linux command or execute an application, it can extract the value of an environment variable
defined in the shell. For example, when I execute the command
archive to archive a file, the
archive command knows where to copy the file based on my shell's value of the variable
which is my personal directory
/admin/username. In this case, the Master module sets a unique
ARCHIVE_HOME for each user, so that one user's files are not archived into another
The last four lines of the module modify an existing shell variable. The first
module statement prepends two directories (/usr/share/man and /usr/local/man) to the variable MANPATH, which
is used in finding
man pages (Linux online documentation). The last three lines each insert
one or two directories at the beginning of the variable
PATH, which is used to find Linux commands
as well as application scripts and executables.
Here is a list of improvements provided by
bcmodule that are not available in
- It translates native modulefile names into BCT-specified modulefile names for greater uniformity of modulefile
names across different systems in the HPCMP. Users can use either the BCT modulefile name or the native modulefile
name interchangeably. Two new sub-commands (tran and show_all_trans) show the translation of modulefile names from
native to BCT and vice-versa, as illustrated below.
$ bcmodule tran compiler/gcc/6.2.0 real module name is gcc/6.2.0 BCT module name is compiler/gcc/6.2.0
- It returns non-zero return codes when errors occur.
- Output from bcmodule goes to stdout, which allows users to pipe the output into subsequent Linux commands
more, as illustrated below.
$ bcmodule avail | grep perftools cray-petsc-complex-64/126.96.36.199 perftools-base/6.3.0 cray-petsc-complex-64/188.8.131.52 perftools-base/6.3.2 cray-petsc/184.108.40.206 perftools/6.4.4This does not work with
modulesince it writes its output to standard error. Each line of output from the bcmodule command above has perftools in it, but it also contains modules that have nothing to do with perftools. This is because
grepis grabbing entire lines. Next we will show a better bcmodule feature for seeing only the perftools related modules.
- A new sub-command
findsearches through all known modulefile directories (not just those in use) and lists all modulefile names it finds that contain the given string searched for, thus helping users find modulefiles when the exact name is uncertain and/or the location is unknown. An illustration of the sub-command
findis included below.
$ bcmodule find perftools /opt/cray/modulefiles/perftools-lite total 36 -rwxr-xr-x 1 root root 7794 Jan 15 2015 6.2.2 -rwxr-xr-x 1 root root 7201 Aug 25 2015 6.2.5 -rw-r--r-- 1 root root 7616 Oct 19 2015 6.3.0 -rw-r--r-- 1 root root 7817 Apr 13 2016 6.3.2 -rw-r--r-- 1 root root 1506 Jan 25 2017 6.4.4 /opt/cray/modulefiles/perftools-base total 28 -rw-r--r-- 1 root root 7707 Oct 19 2015 6.3.0 -rw-r--r-- 1 root root 8015 Apr 13 2016 6.3.2 -rw-r--r-- 1 root root 9707 Jan 25 2017 6.4.4 /opt/cray/modulefiles/perftools total 36 -rwxr-xr-x 1 root root 7794 Jan 15 2015 6.2.2 -rwxr-xr-x 1 root root 7201 Aug 25 2015 6.2.5 -rw-r--r-- 1 root root 7560 Oct 19 2015 6.3.0 -rw-r--r-- 1 root root 7761 Apr 13 2016 6.3.2 -rw-r--r-- 1 root root 1506 Jan 25 2017 6.4.4Here the output format is a bit different, but all the information you need to locate the perftools modules is given. And the timestamps of the files is also given, which is not the case for
- A new sub-command
find_in_modulesearches for a modulefile which contains the given string, thus locating the modulefile not by a string in its name (the sub-command
finddescribed above does that), but by finding a modulefile that performs any action involving the string. For example,
bcmodule find_in_module XYZfinds any modulefile that sets the environment variable XYZ, or that sets any environment variable with XYZ as part of the name, or augments any PATH with a directory having the pattern XYZ, etc.
An illustration of the subcommand
find_in_moduleis included below. Each modulefile listed has the string
ARCHsomewhere in its contents.
$ bcmodule find_in_module ARCH /p/app/modulefiles/unsupported/master /p/app/modulefiles/unsupported/.Master.old /p/app/modulefiles/unsupported/Master /p/app/modulefiles/apps/fluent/172~ /p/app/modulefiles/apps/fluent/180 /p/app/modulefiles/apps/fluent/182
- A new sub-command
inuselists all module directories currently in use.
- A new sub-command
show_all_dirslists all known module directories so that users can easily add module directories for greater access to modulefiles.
- Even easier, the new sub-command
use_all_dirsadds all known module directories so that users automatically use all configured module directories when loading and listing available modules.
Below demonstrates how the number of modules available increases (from 41 lines of output to 100 lines) when all (system configured) modulefile directories are added using the
$ bcmodule avail | wc -l 41 $ bcmodule use_all_dirs $ bcmodule avail | wc -l 100These three commands show that by executing
bcmodule use_all_dirs,the list of module directories known to the shell increases such that an additional 59 lines worth of modules are now locatable, and can thus be loaded. The
show_all_dirssub-commands mentioned above can be used to see those directories.
The term "system configured" used above refers to the commonly used module directories configured for bcmodule by the system admin, but not directories containing only modules for certain projects. Users who wish to always have their project specific module directory used can do so by including a
module usecommand in their personal shell initialization script.
countcounts modulefiles and modulefile directories currently used.
Tired of getting too much output from
module avail? The new bcmodule features
-xd all make
finding modulefiles much easier.
-compressremoves the module version and lists each "family" of modules once instead of once per version. On one system, this reduced the number of lines of output from 202 to 51.
Below includes an example where
module avail, for the directory /opt/modulefiles, gives 21
lines of output whereas the
-compress option in bcmodule shortens the output to just two lines:
$ module avail --------------------------------------- /opt/modulefiles ---------------------------------------- cce/8.3.14(default) intel/220.127.116.11 cce/8.3.5 intel/18.104.22.168 cce/8.3.6 intel/22.214.171.124 cce/8.3.9 intel/126.96.36.199 cce/8.4.0 java/jdk1.7.0_45 cce/8.4.1 java/jdk1.8.0_51(default) cce/8.4.5 modules/188.8.131.52(default) cce/8.5.6 modules/184.108.40.206 chapel/220.127.116.11(default) modules/18.104.22.168 ddt/22.214.171.124_39982(default) pbs ddt/126.96.36.199_42607 pgi/14.10.0 ddt-memdebug/188.8.131.52_39982(default) pgi/14.10.0-acc ddt-memdebug/184.108.40.206_42607 pgi/15.3.0(default) eswrap/1.3.3-1.020200.1278.0(default) pgi/15.3.0-acc gcc/4.8.1 pgi/15.7.0 gcc/4.9.2 pgi/15.7.0-acc gcc/4.9.3(default) pgi/16.10.0 gcc/5.1.0 pgi/16.10.0-acc gcc/5.3.0 xc-sysroot/5.2.40 gcc/6.1.0 xc-sysroot/5.2.82(default) gcc/6.2.0 $ bcmodule avail -compress --------------------------------------- /opt/modulefiles --------------------------------------- cce compiler/gcc compiler/java ddt ddt-memdebug eswrap modules pbs xc-sysroot chapel compiler/intel compiler/pgi
-sf patternlists only modulefiles having
patternin the name.
-xf patternlists only modulefiles not having
patternin the modulefile name.
An illustration of the sub-command
$ bcmodule av -sf perftool ------------------------------------ /opt/cray/modulefiles ------------------------------------- perftools-base/6.3.0 perftools-lite/6.2.5(default) perftools-lite/6.4.4 perftools/6.3.0 perftools-base/6.3.2 perftools-lite/6.3.0 perftools/6.2.2 perftools/6.3.2 perftools-base/6.4.4 perftools-lite/6.3.2 perftools/6.2.5(default) perftools/6.4.4 perftools-lite/6.2.2
-sd dirlists only modulefiles in directories having
diras part of the directory name.
-xd dirlists only modulefiles in directories not having
diras part of the directory name.
The two commands
bcmodule find patternand
module avail –sf patternare very similar, but not identical. Their output formats are different, but more importantly,
findsearches through all configured modulefile directories, whereas the second command only searches through directories you have
The new bcmodule does a slightly better job than module of "cramming" as many columns of output as it can to your screen to reduce the number of lines of output. In the process of reformatting the output, it sometimes alphabetizes the modulefiles slightly differently.