Welcome to the HPC Help Desk Video Series.
This presentation will provide an overview
of the Computational Science Environment
and what tools are available for HPCMP users.
In the link to supplemental materials accompanying this video.
you will find a Job Aid
that includes
all the URLs. phone numbers and email contacts
referenced in this presentation.
When you see this graphic and hear this sound.
it means the information is contained
in the accompanying Job Aid.
This video will first present an overview
of the High Performance Computing Modernization Program
Computational Science Environment. or HPCMP CSE.
Next. we will introduce
the four distinct categories of open source
software available within CSE.
We will take a brief look at applications within each category.
We will explore how to build CSE packages
using GNU or Intel compilers.
Finally. we will introduce the CSE Quick Reference
Guide on the centers.hpc.mil website.
If you are looking for information
about a particular software package.
the timestamp location is shown for each category above.
The HPCMP created the Baseline Configuration Team.
or BCT. to define and establish a common
set of capabilities and functions
so users could compute productively
and collaboratively
when using the HPC resources at multiple computing centers.
The CSE is a collection of tools which combines all BCT-mandated
software and is built in a similar way
on every HPC system in the program.
This allows users to have a common environment
to develop software or have their jobs run in a similar fashion.
independent of the machine.
CSE is comprised of four core categories:
Math/Scientific libraries provide
a wide range of mathematical and scientific
routines. enabling complex computations
and problem solving in parallel and serial environments.
High productivity languages offer general purpose scripting.
numerical arrays. linear algebra.
message passing interfaces and scientific plotting
capabilities. enhancing productivity
and facilitating interactive and parallel computing.
Performance and profiling tools provide
consistent interfaces to hardware counters and events.
scalable trace analysis packages.
performance analysis of parallel
applications and memory management analysis and profiling.
which are essential
for optimizing HPC applications
and ensuring efficient resource utilization.
Utilities covering development and build tools.
source repository revision control. archive and compression tools
and libraries for data models and formats
provide the necessary infrastructure
for efficient software development.
Version control. data archiving and handling of various
data formats in HPC environments.
Let's take a closer look at the Math/Scientific Libraries.
Arpack-NG is a Fortran77 library
designed to solve large scale eigenvalue problems.
Dakota is a flexible software package
that provides an interface between analysis codes
and iterative systems
analysis methods. including optimization
and uncertainty quantification.
FFTW is a C library for computing the discrete Fourier transform
in one or more dimensions of arbitrary
input size and of both real and complex data.
PETSc is another important suite of data structures
and routines
for the scalable solution of scientific applications
modeled by partial differential equations.
SuperLU is a general purpose library
for the direct solution of large. sparse
non-symmetric systems of linear equations.
LAPACK is a software library
that provides routines
for solving systems of linear equations and linear
least squares. eigenvalue problems and singular value
decomposition.
SCALAPACK is a high performance linear algebra library
for parallel distributed memory machines.
solving dense and banded linear systems.
least square problems. eigenvalue problems
and singular value problems.
Lastly. GSL. the GNU Scientific Library. is a software library
for numerical computations
in applied mathematics and science.
written in C and accessible from other programming languages.
CSE incorporates a number of high productivity
languages. offering both general purpose
and specialized scripting applications.
Python has a general purpose scripting language.
Its simplicity and readability
make it a popular choice
among programmers and researchers alike.
Python plays a pivotal role in high performance computing.
providing a comprehensive
and powerful environment
for scientific computation and data analysis.
NumPy provides support for large. multi-dimensional
arrays and matrices.
which are fundamental data structures
in scientific computation.
It also offers a collection of mathematical functions
for numerical computations.
SciPy. built on NumPy. is an ecosystem of open source software
for mathematics. science and engineering.
It allows users to manipulate
and visualize data with a wide range of high level commands.
For distributed computing.
Python incorporates the message
passing interface standard with programs
such as PyMPI and MPI4Py.
These tools
allow Python programs to leverage multiple
processors. thereby enhancing their performance
and enabling them to handle larger.
more complex computational tasks.
IPython. an interactive command line terminal
for Python.
offers an enhanced read-eval-print loop environment
that is particularly well-suited to scientific computing.
It allows for dynamic code execution.
which is essential
for high performance computing.
where computations can be complex and resource intensive.
Octave. a high level interpretive language primarily intended
for numerical computations.
is often considered a free alternative to MATLAB.
It provides capabilities for the numerical solution
of linear
and nonlinear problems
and has extensive graphics capabilities for data visualization.
Lastly.
Matplotlib. a plotting library
for Python and its numerical mathematics extension
NumPy. provides an object-oriented API
for embedding plots into applications.
This is particularly useful in high performance
computing where visualizing data can help
in understanding complex patterns
and relationships In the data.
Performance
analysis is crucial for parallel applications
as it helps in identifying bottlenecks.
understanding the behavior of applications.
and consequently improving their efficiency.
These CSE tools
play a significant role in the development
and optimization of parallel applications.
PAPI. Performance Application Programming Interface provides
a consistent and standard interface
to hardware counters and events.
It's a critical tool for developers
because it allows them to access low level
performance details directly from the CPU.
This can help in identifying performance issues
that might not be visible
at the software level.
such as cache misses or branch mispredictions.
Scalasca. or Scalable Trace
Analysis Package is designed to analyze
the runtime behavior of parallel
programs. essential in identifying
performance hotspots and understanding their causes.
Scalasca can scale itself to handle
large code bases and complex parallel execution environments.
making it a valuable tool for performance
optimization and high performance computing.
Tau. Tuning and Analysis Utilities.
is a comprehensive
profiling and tracing toolkit for performance
analysis of parallel programs.
Tau supports a wide range of programming languages
and parallel execution models.
It provides detailed information about program execution.
including time spent in
routines. message passing details
and hardware counter performance metrics.
This level of detail is crucial
for optimizing parallel applications.
where performance issues
can often be hidden in the complexity of parallel execution.
Valgrind is a powerful tool for memory management
analysis and profiling.
It can automatically detect
many memory management and threading bugs.
which are common issues
In parallel programming.
CSE includes a variety of utilities and systems
that collectively support the complex processes
involved in high performance computing.
These can be broadly categorized
into development and build tools.
source repository/revision control
systems. archive and compression
utilities. and. libraries for data models and formats.
Development and build tools include
autoconf. automake. Bison. Boost C++ and CMake.
These tools are integral to the software
development lifecycle.
facilitating the creation of applications that are portable.
maintainable and robust.
They provide a framework for producing shell scripts.
generating Makefile.in files.
converting grammar descriptions into C programs.
enhancing the functionality of C++
with a collection of libraries.
and managing the build process
of software using compiler-independent methods.
Complementing these are several source repository/
revision control systems.
GNU cvs is a version control system
that documents
the history of sources files. Subversion (or svn) is a software
versioning and revision control system
that allows you to track changes to source code and files.
Git is a distributed version control
system designed to handle everything
from small to very large projects efficiently.
Archive and compression
utilities are used to make files easier to manage and transfer.
GNU tar is a utility for collecting many files into one.
GNU gzip and gunzip
are used for file compression and decompression.
Also. bzip2 and bunzip2 are open
source file compression programs
that use the Burrows-Wheeler algorithm.
Libraries for data
models and formats provide
standardized formats for storing and managing data.
HDF5 is a versatile data model
that can represent very complex data
objects and a wide variety of metadata.
NetCDF is a set of software libraries
and machine-independent data formats
that support the creation. access
and sharing of array-oriented scientific data.
XDMF is a standardized way
to access
data produced by high performance
computing codes. widely
used in the scientific simulation community.
All packages and libraries
in CSE are built on each
HPC system with the GNU compiler collection.
The most recent
GNU compiler version is loaded by default
from the module command.
A subset of the CSE applications
is also built with the Intel compiler.
Information on Intel-specific
builds of CSE can be found
within the CSE Quick Reference Guide.
In special cases
where there is need to build a portion
or all of the CSE package with an alternate compiler.
users are asked to place a request at the
HPC Help Desk. Information on contacting the HPC Help Desk
as contained in the Job Aid included with this presentation.
The initial setup involves loading the CSE init module.
The available software can be viewed using module avail cse
The module name format is cse/application/version
with the last tag
referring to the desired version of the application.
If the version tag is omitted
when loading modules.
the newest version of the software is loaded
Regardless of whether you build the modules
with the CSE init version or use the Intel compiler.
module names for CSE
software remain the same. Information on all CSE packages
and libraries. including module commands for access.
can be found in the CSE Quick Reference Guide
located on the centers.hpc.mil website.
The various titles covered in
this presentation are not an exhaustive list
of applications within the CSE suite.
You are encouraged to visit the Quick Reference
Guide for all available software.
Also. if you have suggestions for other applications
that should be included
in the Computational Science Environment.
please make suggestions to the Baseline Configuration Team.
Contact information is available in the accompanying Job Aid.
By using open source
software. users can leverage
the collective intelligence of a global community
of developers and researchers.
This can lead to more robust.
efficient and secure software
as issues can be identified
and addressed quickly by the community.
Furthermore. open source software provides transparency.
allowing users to understand
exactly how computations are performed.
which is critical in research and development contexts
where reproducibility is key.
in the context of the CSE software and libraries.
adherence to open source policies ensures
that the software stack remains accessible. adaptable
and up-to-date.
It's important
to remember that the strength of open source lies
in its collaborative nature.
Every user is also a potential contributor.
Thank you for taking the time to view this instructional video.
If you have any questions or experience any problems.
please contact the HPC Help Desk.
The phone numbers.
email address and HPC Help Desk Service Portal
URL information are shown here
and can also be found on the accompanying Job Aid.