Welcome to the HPC Help Desk Video Series.
In this video we introduce you to Modules and demonstrate
module commands, so you are encouraged to read the
Modules documentation available at each center's website.
If you want to follow along, you will need to have the DoD Kerberos
kit installed, with a Kerberized version of PuTTY or Open SSH
on your machine and the ability to log into an HPC system.
In the link to supplemental materials accompanying this video,
you will find a Job Aid, including all the URLs, phone numbers and
email contacts referenced in this video.
We will begin by explaining what modules are.
Next, we will demonstrate various module commands including:
viewing available modules and currently loaded modules,
how to load modules, how to remove loaded modules,
and how to swap a loaded module with a different module.
And last, module conflicts will also be addressed.
Not all of the HPC systems have the same modules,
and the systems can contain a wide range of modules
due to diverse and extensive software residing on these systems.
What are modules?
Modules provide an automated way to manage your
software environment. Without modules, you would need to know
many specifics about the software you wish to use such as:
where the software is installed, dependencies, and
application-specific environment variables.
The availability of modules allows these specifications to be
transparent to you. To use most Commercial-Off-The-Shelf,
commonly referred to as COTS, Government-Off-The-Shelf
also known as GOTS, and even open source software
requires loading modules for that software.
Modules also make it easy for you to control what version
of a particular software package should be used by your job.
The module command makes use of modulefiles to control your
software environment. These modulefiles are essentially scripts
executed by the module command. The modulefiles specify how
your software environment must be modified. This includes
changes to the system environment variables,
such as $PATH and $LD_LIBRARY_PATH,
as well as possibly application-specific environment variables,
such as $MPI_HOME.
The modulefile may load additional modules
to enforce dependencies.
Finally, the modulefile may specify conflicts, such as when
one module conflicts with another,
or both modules cannot be loaded simultaneously.
In this section we introduce the module command
and demonstrate subcommands available for use with it.
We will begin by obtaining a Kerberos ticket and then
logging into the Mustang HPC system.
The command associated with the modules is called module,
and does not function by itself.
The full format of the modules command is shown here.
While this introductory video focuses on module sub-commands
and modulefiles, we will briefly describe
the first parameter the module command accepts: switches.
For demonstration, we enter the HELP switch.
There are two formats this can take:
a single dash and uppercase H, or two dashes and the word
“help” in lower case
The result is a list of accepted command-line switches, all of them
showing a single letter form or the full word notation.
Most of the command-line switches work only with
specific module subcommands.
For example the switches “terse” and “long”
apply just to the “avail” and “list” subcommands
and control the display of the output in either
short or long format respectively.
The list of switches is followed by a list of available sub-commands.
For more extensive documentation than the HELP switch provides,
you can view the module manual by entering “man module”
at the command prompt.
Scroll down and a list of the switches and detailed explanation
of how they are used is found, as shown here.
Following the list of switches are detailed descriptions
of the subcommands and the arguments they require.
To exit the manual, press “q” on your keyboard.
Now, let us focus on the subcommands,
which define the action to take place.
We will demonstrate the following sub-commands:
list, avail, load, unload, swap or switch, show or display
We begin by viewing your currently loaded modulefiles.
To view, use the subcommand list.
The output will be a numbered list of all your loaded modulefiles
at the instant the command is run.
Each HPC system has its own set of pre-loaded modulefiles.
An example is shown here.
Most systems load a minimal number of default modules at login.
In this example the default compiler module loaded is
Intel/18.0.1.163.
This can be changed as we will see shortly.
To see the available modulefiles on the system you are logging into,
you use the subcommand avail.
The result of this command will be a long listing of all modules
you can use on the system,
including version numbers in their name.
The modules are generally grouped in module directories
based on use.
For example, a “devel” directory for software typically used
in software development
such as compilers, debuggers, and message passing libraries.
Within these directories, the listing is not numbered,
but it is alphabetized.
If one module name has multiple versions, you will likely see a label
next to one of them indicating which one
is the default module version.
For example, there are four versions of the
GNU compiler collection modules listed here,
and version 7.3.0 is shown as the default version.
You can also filter the list by providing a filter to the module avail.
For example, if you wanted to see only the compilers available,
you can type “module avail compiler” and the result is shown here.
Next, we demonstrate how to load and unload modules.
As mentioned previously, modules allow you to easily manage
your software environment.
For this example we want to use the software package Matlab.
To confirm if it is available, use the which command
We see Matlab is not found.
This is because the location of the executable is not in our
$PATH environment variable.
To see if Matlab is available via modules,
we will use the module avail command.
If there is a default version of the software, it will be notated.
To load the default version, simply enter the module load command
followed by the software name: in this case module load Matlab
A module list shows the default version of the Matlab module
was loaded.
The location of that Matlab executable has been prepended
to our $PATH environment variable,
shown here, and Matlab is now available from the command-line.
Unloading modules is very simple. Enter module unload
followed by the software.
In this example, the command is module unload Matlab
As we can see, the Matlab module has been removed from our list.
In addition, the Matlab binary location has been removed
from the $PATH environment variable.
But what if you have a requirement to use a version
other than the default?
Executing the "avail" command, the result shows
there are two other versions of Matlab.
You can load any version by providing the full modulefile name,
including the trailing version number.
To use this version of Matlab, enter module load matlab/R2017b.
Using module list, the module for the older version of Matlab
is now loaded.
There is a command to swap a module out
and replace it with another.
Suppose there is already a module loaded for a compiler,
but it is either not the compiler you wish to use,
or it is not the version you need for your work
This is not uncommon since a default compiler may be loaded
into your environment at login.
In this example, an Intel compiler is loaded,
while we have a requirement to use the Gnu compiler suite.
You can use the commands already covered and unload the current
version followed by loading the desired version.
However, you can obtain the same result with a single command.
The module swap command takes two arguments,
the module file to be unloaded,
followed by the module file to be loaded.
For this example it is simply module swap compiler/intel gcc.
Since there is only a single compiler loaded at the start,
we do not need to specify the version
for the module being swapped out.
Here we see the Intel compiler is no longer available,
and the Gnu compiler suite is now available.
As a reminder, the swap and switch subcommands
are interchangeable,
and you will see both in the various Modules User Guides.
Also, the show and display subcommands,
reviewed in the next section, are interchangeable.
What happens when a module conflicts with another?
Some modules are incompatible with other modules.
To have both loaded simultaneously
can have erroneous or unknown consequences.
This is often true for compilers,
but can also hold for message passing libraries and others.
The modules software specifies these conflicts
and will prohibit a conflicting module from loading.
To determine if a module will conflict with another module,
use the subcommand show or display.
Here we have a module show compiler/intel
The output of this command tells you what happens when you load
the module you are showing, and it can be extensive.
Immediately beneath is the specific version
of the intel compiler loaded.
The keyword you should look for is “conflict”
and will be on the left-hand side of the list,
commonly towards the top.
We see a conflict is specified for other intel versions
and gcc compilers.
When we try to load one of those conflicting modules
without unloading the current one,
you will see a short error message informing you it is conflicting
with a current module and
will not load. If you need to use this other module,
you must unload the currently loaded module
using techniques previously reviewed.
This has only been an introduction to the use
of modules on HPCMP resources.
We hope this has been sufficient to get you started.
Modules can be very powerful and provide great value.
We encourage you to learn more through other sources,
including the modules user guides available
from each center’s Documentation website,
as shown here.
Thank you for taking the time to view this instructional video.
If you have any questions or experience any problems,
please contact the HPC Help Desk.
The phone numbers, email address,
and HPC Help Desk Web Portal URL information
is shown here and can also be found on the accompanying job aid.