1. Introduction

1.1. Document Scope and Assumptions

This document provides an overview and introduction to the use of the Penguin Computing TrueHPC (Raider) located at the AFRL DSRC, along with a description of the specific computing environment on the system. The intent of this guide is to provide information that will enable the average user to perform computational tasks on the system. To receive the most benefit from the information provided here, you should be proficient in the following areas:

• Use of the Linux operating system
• Use of an editor (e.g., vi or emacs)
• Remote use of computer systems via network
• A selected programming language and its related tools and libraries

1.2. DSRC Policies

All policies are discussed in the Policies Section of the AFRL DSRC Introductory Site Guide. All users running at the AFRL DSRC are expected to know, understand, and follow the policies discussed. If you have any questions about the AFRL DSRC's policies, please contact the HPC Help Desk.

1.3. Obtaining an Account

To begin the account application process, visit the Obtaining an Account page and follow the instructions presented there. An HPC Help Desk video is available to guide you through the process.

1.4. Training

Training on a number of topics in this User Guide is available at the PET Knowledge Management Learning System. New account holders should strongly consider attending HPCMP New Account Orientation, which is provided via live webcast every month and available as an on-demand video.

1.5. Requesting Assistance

The HPC Help Desk is available to assist users with unclassified problems, issues, or questions. Technicians are on duty 8:00 a.m. to 8:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).

• Service Portal: https://helpdesk.hpc.mil/hpc
• E-mail: help@helpdesk.hpc.mil
• Phone: 1-877-222-2039 or 937-255-0679

For more information about requesting assistance, see the HPC Help Desk dropdown.

2. System Configuration

2.1. System Summary

Raider is a Penguin Computing TrueHPC system. It has ten login nodes Node - an individual server in a cluster or collection of servers of an HPC system and five types of compute nodes for job execution. Raider uses InfiniBand as its high-speed interconnect Interconnect - a specialized, very high-speed network that connects the nodes of an HPC system together. It is typically used for application inter-process communication (e.g., message passing) and I/O traffic. for MPI messages and IO traffic. Raider uses Lustre to manage its parallel file system Parallel File System - A software component designed to store data across multiple networked servers and to facilitate high-performance access through simultaneous, coordinated input/output operations (IOPS) between clients and storage nodes..

2.2. Login and Compute Nodes

Raider is intended as a batch-scheduled Batch-scheduled - users request compute nodes via commands to batch scheduler software and wait in a queue until the requested nodes become available HPC system with numerous nodes. Its login nodes Login Node - a node that serves as the user's entry point into an HPC system are for minor setup, housekeeping, and job preparation tasks and are not used for large computational (e.g., memory, IO, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes Compute Node - a node that performs computational tasks for the user. There may be multiple types of compute nodes for specialized purposes. by batch job Batch Job - a single request for a set of compute nodes along with a set of tasks (usually in the form of a script) to perform on those nodes submission. Node types such as "Standard", "Large-Memory", "GPU-Accelerated", etc. are considered compute nodes. Raider uses both shared Shared Memory Model - a programming methodology where a set of processors (such as the cores within one node) have direct access to a shared pool of memory and distributed Distributed Memory Model - a programming methodology where memory is distributed across multiple nodes giving processes on each node faster direct access to local memory, but requiring slower techniques such as message passing to access memory on other nodes memory models. Memory is shared among all the cores on one node but is not shared among the nodes across the cluster.

Raider's ten login nodes use AMD 7713 Milan processors with 503 GB of usable memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not use more than 8 GB of memory at any one time. Login nodes raider01-raider06 are each paired with one NVIDIA A40 GPU, with 45 GB of usable memory. Login nodes raider07-raider10 (also known as viz-login nodes) are paired one NVIDIA A100 GPU, with 40 GB of usable memory.

Raider's standard compute nodes use AMD 7713 Milan processors. Each node contains 251 GB of usable shared memory. Standard compute nodes are intended for typical compute jobs.

Raider's large-memory compute nodes use AMD 7713 Milan processors. Each node contains 2.0 TB of usable shared memory. Each also contains 7.68 TB of on-node NVMe SSD storage. Large-memory compute nodes are intended for jobs requiring large amounts of memory.

Raider's visualization nodes consist of AMD 7713 Milan processors paired with one NVIDIA A40 PCIe 4 GPU. Each node contains 503 GB of usable shared memory on the node, as well as 48 GB of shared memory internal to each accelerator. Visualization compute nodes are intended for hardware accelerated graphics.

Raider's MLA nodes consist of AMD 7713 Milan processors paired with four NVIDIA A100 SXM 4 GPUs. Each node contains 503 GB of usable shared memory on the node, as well as 40 GB of shared memory internal to each accelerator. Each node also contains 3.84 TB of on-node NVMe SSD storage. MLA compute nodes are intended for intensive GPU applications such as machine learning and data analytics.

Raider's High core performance, or High Clock nodes use AMD 73F3 Milan processors. Each node contains 503 GB of usable shared memory. High core performance nodes are intended for jobs using enterprise software licensed per core and applications that do not scale well.

3. Accessing the System

3.1. Kerberos

For security purposes, you must have a current Kerberos Kerberos - authentication and encryption software required by the HPCMP to access HPC system login nodes and other resources. See Kerberos & Authentication ticket on your computer before attempting to connect to Raider. To obtain a ticket you must either install a Kerberos client kit on your desktop or connect via the HPC Portal. Visit the Kerberos & Authentication page for information about installing Kerberos clients on your Windows, Linux, or Mac desktop. Instructions are also available on those pages for getting a ticket and logging into the HPC systems from each platform.

3.2. Logging In

The system host name for the Raider cluster is raider.afrl.hpc.mil, which redirects you to one of 10 login nodes. Hostnames and IP addresses to these nodes are available upon request from the HPC Help Desk.

The preferred way to login to Raider is via ssh, as follows: % ssh username@raider.afrl.hpc.mil

3.3. File Transfers

File transfers to DSRC systems (except for those to the local archive system) must be performed using the following HPCMP Kerberized tools: scp or sftp. Windows users may use a graphical secure file transfer protocol (sftp) client such as FileZilla. See the HPC Help Desk Video on Using FileZilla. Before using any of these tools, you must use a Kerberos client to obtain a Kerberos ticket. Information about installing and using a Kerberos client can be found on the Kerberos & Authentication page.

Note: mpscp is no longer supported and will be removed from future releases of HPCMP Kerberos. Users should discontinue use of mpscp.

The command below uses secure copy (scp) to copy a single local file into a destination directory on a Raider login node. % scp local_file username@raider.afrl.hpc.mil:/target_dir

The scp command can be used to send multiple files. This command transfers all files with the .txt extension to the same destination directory. % scp *.txt username@raider.afrl.hpc.mil:/target_dir

The example below uses the secure file transfer protocol (sftp) to connect to Raider, then uses sftp's cd and put commands to change to the destination directory and copy a local file there. The sftp quit command ends the sftp session. Use the sftp help command to see a list of all sftp commands. % sftp username@raider.afrl.hpc.mil sftp> cd target_dir sftp> put local_file sftp> quit

4. User Environment

4.1. User Directories

The following user directories are provided for all users on Raider:

File Systems on Raider

Path	Formatted Capacity	File System Type	Storage Type	User Quota	Minimum File Retention
/p/home ($HOME)	1.3 PB	Lustre	HDD	30 GB	None
/p/work1 ($WORKDIR)	18 PB	Lustre	Hybrid SSD/HDD	None	30 Days
/p/work2 ($WORKDIR2)	820 TB	Lustre	Hybrid SSD/HDD	None	30 Days
/p/cwfs ($CENTER)	3.3 PB	GPFS	HDD	100 TB	120 Days
/p/app ($PROJECTS_HOME)	689 TB	Lustre	NVMe SSD	None	None

4.1.1. Home Directory ($HOME)

When you log in, you are placed in your home directory, /p/home/username. It is accessible from the login and compute nodes and can be referenced by the environment variable $HOME.

Your home directory is intended for storage of frequently used files, scripts, and small utility programs. It has a 30-GB quota, and files stored there are not subject to automatic deletion based on age. It is backed up weekly to enable file restoration in the event of catastrophic system failure.

Important! The home file system is not tuned for parallel I/O and does not support application-level I/O. Jobs performing intensive file I/O in your home directory will perform poorly and cause problems for everyone on the system. Running jobs should use the work file system ($WORKDIR) for file I/O.

4.1.2. Work Directory ($WORKDIR)

The work file system is a large, high-performance Lustre-based file system tuned for parallel application-level I/O. It is accessible from the login and compute nodes and provides temporary file storage for queued and running jobs.

All users have a work directory, /p/work1/username, on this file system, which can be referenced by the environment variable, $WORKDIR. This directory should be used for all application file I/O. NEVER allow your jobs to perform file I/O in $HOME.

$WORKDIR has no quota. It is not backed up or exported to any other system and is subject to an automated deletion cycle. If available disk space gets too low, files that have not been accessed in 30 days may be deleted. If this happens or if catastrophic disk failure occurs, lost files are irretrievable. To prevent the loss of important files, transfer them to a long-term storage area, such as your archival directory ($ARCHIVE_HOME, see Archive Usage), which has no quota. Or, for smaller files, your home directory ($HOME).

Maintaining the high performance and stability of the Lustre file system is important for the efficient and effective use of Raider by all users. For example, setting stripe counts can maximize your performance and prevent you from filling up a single file system component causing system instability.

To avoid errors that can arise from two jobs using the same scratch directory, a common technique is to create a unique subdirectory for each batch job. See Sample Scripts for an example of a script that does this.

4.1.3. Additional Work Directory ($WORKDIR2 or /p/work2)

This system includes an additional specialized work directory, $WORKDIR2, which is an NVMe-based file system. SSD-based Lustre file systems perform well for all I/O use cases, but they have a major advantage over HDDs for many small file or random file accesses. All users have a $WORKDIR2 directory, /p/work2/username.

$WORKDIR2 has no quota. It is not backed up or exported to any other system and is subject to an automated deletion cycle. If available disk space gets too low, files that have not been accessed in 30 days may be deleted.

4.1.4. Center Directory ($CENTER)

Note: Raider is not yet accessible from the AFRL HPC Portal. Users will be notified as soon as they can access the system via the Portal.

The Center-Wide File System (CWFS) is an NFS-mounted file system. It is accessible from the login nodes of all HPC systems at the center and from the HPC Portal. It provides centralized, shared storage that enables users to easily access data from multiple systems. The CWFS is not tuned for parallel I/O and does not support application-level I/O.

All users have a directory on the CWFS. The name of your directory may vary between systems and between centers, but the environment variable $CENTER always refers to this directory.

$CENTER has a quota of 100 TB. It is not backed up or exported to any other system and is subject to an automated deletion cycle. If available disk space gets too low, files that have not been accessed in 120 days may be deleted. If this happens or if catastrophic disk failure occurs, lost files are irretrievable. To prevent the loss of important files, transfer them to a long-term storage area, such as your archival directory ($ARCHIVE_HOME, see Archive Usage), which has no quota. Or, for smaller files, your home directory ($HOME).

4.1.5. Projects Directory ($PROJECTS_HOME)

The Projects directory, $PROJECTS_HOME, is a file system set aside for group-shared storage. It is intended for storage of semi-permanent files, similar to a home directory, but typically larger and shared by a group. It is not meant for high-speed application output ($WORKDIR, see Work Directory). A new project sub-directory can be created via an HPC Help Desk request and appears as follows: $PROJECTS_HOME/new_group_dir. The HPC Help Desk request must specify a UNIX group to be assigned to the project sub-directory. Users can create and manage UNIX groups in the Portal to the Information Environment, allowing the creator of the assigned group to manage the members of the group with access to the project sub-directory.

4.1.6. Storage On-node ($LOCALWORKDIR)

Some compute nodes may include a local solid-state storage device (NVMe SSD) that is local to and accessible by the node only and can be accessed by the environment variable $LOCALWORKDIR. It has improved local bandwidth and latency, but each device is a separate drive with no parallel read/write capability. Files stored on this device must be relocated at the end of a job or they may be lost when the node is reassigned to a new job.

NOTE: $LOCALWORKDIR is currently undefined on Raider, and users are not able to access on-node storage at this time.

4.1.7. Specialized Temporary Directories

Each node includes several specialized directories.

The /tmp and /var/tmp directories are usually intended for temporary files as created by the operating system. Do not use these directories for your own files, as filling up these file systems can cause issues.

Raider also provides a "virtual" file system (i.e., "RAM disk") called /dev/shm which is local to each compute node. You may use this file system to store files in memory. It automatically increases in size as needed, up to half of the memory of the node. It is extremely fast, but it is also small and takes available node memory away from your application. An example use case is performing significant I/O with many small files when the memory is not otherwise needed by the application.

4.2. Shells

The following shells are available on Raider: csh, bash, ksh, tcsh, sh, and zsh.

To change your default shell, log into the Portal to the Information Environment and go to "User Information Environment" > "View/Modify personal account information". Scroll down to "Preferred Shell" and select your desired default shell. Then scroll to the bottom and click "Save Changes". Your requested change should take effect within 24 hours.

4.3. Environment Variables

A number of environment variables are provided by default on all HPCMP high performance computing (HPC) systems. We encourage you to use these variables in your scripts where possible. Doing so will help simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems.

4.3.1. Common Environment Variables

The following environment variables are automatically set in both your login and batch environments:

Common Environment Variables

Variable	Description
$ARCHIVE_HOME	Your directory on the archive system
$ARCHIVE_HOST	The host name of the archive system
$BC_ACCELERATOR_NODE_CORES	The number of CPU cores per node for a compute node which features CPUs and a hosted accelerator processor
$BC_BIGMEM_NODE_CORES	The number of cores per node for a big memory (BIGMEM) compute node
$BC_CORES_PER_NODE	The number of CPU cores per node for the node type on which the variable is queried
$BC_HOST	The generic (not node specific) name of the system. Examples include centennial, mustang, onyx and gaffney
$BC_NODE_TYPE	The type of node on which the variable is queried. Values of $BC_NODE_TYPE are: LOGIN, STANDARD, PHI, BIGMEM, BATCH, or ACCELERATOR
$BC_PHI_NODE_CORES	The number of Phi cores per node, if the system has any Phi nodes. It will be set to 0 on systems without Phi nodes
$BC_STANDARD_NODE_CORES	The number of CPU cores per node for a standard compute node
$CC	The currently selected C compiler. This variable is automatically updated when a new compiler environment is loaded
$CENTER	Your directory on the Center-Wide File System (CWFS)
$CSE_HOME	The top-level directory for the Computational Science Environment (CSE) tools and applications
$CXX	The currently selected C++ compiler. This variable is automatically updated when a new compiler environment is loaded
$DAAC_HOME	The top level directory for the DAAC (Data Analysis and Assessment Center) supported tools
$F77	The currently selected Fortran 77 compiler. This variable is automatically updated when a new compiler environment is loaded
$F90	The currently selected Fortran 90 compiler. This variable is automatically updated when a new compiler environment is loaded
$HOME	Your home directory on the system
$JAVA_HOME	The directory containing the default installation of JAVA
$KRB5_HOME	The directory containing the Kerberos utilities
$LOCALWORKDIR	A high-speed work directory that is local and unique to an individual node, if the node provides such space
$PET_HOME	The directory containing tools installed by PET staff, which are considered experimental or under evaluation. Certain older packages have been migrated to $CSE_HOME, as appropriate
$PROJECTS_ARCHIVE	The directory on the archive system in which user-supported applications, code, and data may be kept
$PROJECTS_HOME	The directory in which user-supported applications and codes may be installed
$SAMPLES_HOME	A directory that contains the Sample Code Repository, a variety of sample codes and scripts provided by a center's staff
$WORKDIR	Your work directory on the local temporary file system (i.e., local high-speed disk)

4.3.2. Batch-Only Environment Variables

In addition to the variables listed above, the following variables are automatically set only in your batch environment. That is, your batch scripts can see them when they run. These variables are supplied for your convenience and are intended for use inside your batch scripts.

Batch-Only Environment Variables

Variable	Description
$BC_MEM_PER_NODE	The approximate maximum memory (in integer MB) per node available to an end user program for the compute node type to which a job is being submitted
$BC_MPI_TASKS_ALLOC	The number of MPI tasks allocated for a particular job
$BC_NODE_ALLOC	The number of nodes allocated for a particular job
$JOBDIR	Job-specific directory in $WORKDIR immune to scrubbing while job is active

Please refer to the Raider Slurm Guide for a number of helpful environment variables provided during batch runs.

4.4. Archive Usage

All our HPC systems have access to an online archival mass storage system that provides long-term storage for users' files on a petascale tape file system that resides on a robotic tape library system. A 100-TB disk cache frontends the tape file system and temporarily holds files while they are being transferred to or from tape.

Tape file systems have very slow access times. The tapes must be robotically pulled from the tape library, mounted in one of the limited number of tape drives, and wound into position for file archival or retrieval. For this reason, users should always tar up their small files in a large tarball when archiving a significant number of files. A good size range for tarballs is about 500 GB - 1 TB. At that size, the time required for file transfer and tape I/O is reasonable. Files larger than 10 TB will greatly increase the time required for both archival and retrieval. Files larger than 19 TB will not be archived.

The environment variable $ARCHIVE_HOME is automatically set for you and can be used to reference your archive directory when using archive commands.

4.4.1. Archive Command Synopsis

A synopsis of the archive utility is listed below. For information on additional capabilities, see the AFRL DSRC Archive Guide or read the online man page available on each system. The archive command is non-Kerberized and can be used in batch submission scripts if desired.

Copy one or more files from the archive system: archive get [-C path] [-s] file1 [file2...]

List files and directory contents on the archive system: archive ls [lsopts] [file/dir ...]

Create directories on the archive system: archive mkdir [-C path] [-m mode] [-p] [-s] dir1 [dir2 ...]

Copy one or more files to the archive system: archive put [-C path] [-D] [-s] file1 [file2 ...]

5. Program Development

5.1. Modules

Software modules are a convenient way to set needed environment variables and include necessary directories in your path so commands for particular applications can be found. Raider also uses modules to initialize your environment with application software, system commands, libraries, and compiler suites.

A number of modules are loaded automatically as soon as you log in. To see the currently loaded modules, use the module list command. To see the entire list of available modules, use the module avail command. You can modify the configuration of your environment by loading and unloading modules. For complete information on how to do this and other information on using modules, see the AFRL DSRC Modules Guide.

5.2. Programming Models

Raider supports several parallel programming models. A programming model augments a programming language with parallel processing capability. Different programming models may use a different approach to express parallelism, such as message passing, threads, distributed memory, shared memory, etc.

Note, if an application is not programmed for distributed memory, then only the cores on a single node can be used. This is limited to 128 cores on Raider's standard nodes. See the Node Configuration table for core counts on other nodes.

Note, keep the system architecture in mind during code development. For instance, if your program requires more memory than is available on a single node, then you need to parallelize your code so it can function across multiple nodes.

Key supported programming models are discussed in each subsection below.

5.2.1. Message Passing Interface (MPI)

Raider's default MPI stack supports the MPI 4.0 Standard. MPI is part of the software support for parallel programming across a network of computer systems through a technique known as message passing. MPI establishes a practical, portable, efficient, and flexible standard for high-performance message passing. See man intro_mpi for additional information.

When creating an MPI program, ensure the default MPI module (penguin/openmpi/4.1.4/aocc) or other available MPI module (mpich, mvapich2, Intel/mpi) is loaded. To check this, run the module list command. To load the desired module, run the following command: module load penguin/openmpi/4.1.4/aocc

Also, ensure the source code contains one of the following for the MPI library:

INCLUDE "mpif.h"        ## for older Fortran
USE mpi                 ## for newer Fortran
#include <mpi.h>        ## for C/C++

To compile an MPI program, use one of the following:

mpif90 -o MPI_executable mpi_program.f       ## for Fortran
mpicc -o MPI_executable mpi_program.c        ## for C
mpiCC -o MPI_executable mpi_program.cpp      ## for C++

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

To run an MPI program within a batch script, load the same modules as used to compile the application before using the following command to launch your executable: mpiexec -n mpi_procs ./MPI_executable [user_arguments] where mpi_procs is the number of MPI processes being started. For example: #### The following starts 128 MPI processes #### (the placement of the processes on nodes is handled by the batch scheduler) mpiexec 128 ./MPI_executable

For more information about mpiexec, type man mpiexec.

For more information on which MPI Standard features are supported by the default MPI on the system, check the BC MPI Test Suite page.

5.2.2. Open Multi-Processing (OpenMP)

OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications. It supports shared-memory multiprocessing programming in C, C++, and Fortran and consists of a set of compiler directives, library routines, and environment variables that influence compilation and run-time behavior.

When creating an OpenMP program, if using OpenMP functions (e.g., omp_get_wtime), ensure the source code includes one of the following lines:

INCLUDE "omp.h"        ## for older Fortran
USE omp_lib            ## for newer Fortran
#include <omp.h>       ## for C/C++

To compile an OpenMP program, ensure the desired compiler module is loaded. Use the following compiler commands and flags:

flang -o OpenMP_executable -fopenmp openmp_program.f        ## for AOCC Fortran
ifort -o OpenMP_executable -qopenmp openmp_program.f        ## for Intel Fortran
gfortran -o OpenMP_executable -fopenmp openmp_program.f     ## for GNU Fortran
nvfortran -o OpenMP_executable -mp openmp_program.f         ## for NVIDIA Fortran

clang -o OpenMP_executable -fopenmp openmp_program.c        ## for AOCC C
icc -o OpenMP_executable -qopenmp openmp_program.c          ## for Intel C
gcc -o OpenMP_executable -fopenmp openmp_program.c          ## for GNU C
nvc -o OpenMP_executable -mp openmp_program.c               ## for NVIDIA C

clang++ -o OpenMP_executable -fopenmp openmp_program.cpp    ## for AOCC C++
icpc -o OpenMP_executable -qopenmp openmp_program.cpp       ## for Intel C++
g++ -o OpenMP_executable -fopenmp openmp_program.cpp        ## for GNU C++
nvc++ -o OpenMP_executable -mp openmp_program.cpp           ## for NVIDIA C++

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

When running OpenMP applications, the $OMP_NUM_THREADS environment variable must be used to specify the number of threads. For example: #### run 32 threads on one node export OMP_NUM_THREADS=32 ./OpenMP_executable [user_arguments]

In the example above, the application starts the OpenMP_executable on one node and spawns a total of 32 threads. Since Raider has 128 cores per compute node, if you wanted to run one thread per core, you would set $OMP_NUM_THREADS to 128 instead.

5.2.3. Hybrid MPI/OpenMP

An application built with the hybrid model of parallel programming can run using both OpenMP and Message Passing Interface (MPI). This allows the application to run on multiple nodes yet leverages OpenMP's advantages within each node. In hybrid applications, multiple OpenMP threads are spawned by MPI processes, but MPI calls should not be issued from OpenMP parallel regions or by an OpenMP thread.

When creating a hybrid MPI/OpenMP program, follow the instructions in both the MPI and OpenMP sections above for creating your program.

To compile a hybrid program, use the MPI compilers in conjunction with the OpenMP options, as follows:

mpif77 -o hybrid_executable -fopenmp hybrid_program.f        ## for AOCC Fortran
mpif77 -o hybrid_executable -qopenmp hybrid_program.f        ## for Intel Fortran
mpif77 -o hybrid_executable -fopenmp hybrid_program.f        ## for GNU Fortran
mpif77 -o hybrid_executable -mp hybrid_program.f             ## for NVIDIA Fortran
          
mpicc -o hybrid_executable -fopenmp hybrid_program.c         ## for AOCC C
mpicc -o hybrid_executable -qopenmp hybrid_program.c         ## for Intel C
mpicc -o hybrid_executable -fopenmp hybrid_program.c         ## for GNU C
mpicc -o hybrid_executable -mp hybrid_program.c              ## for NVIDIA C
          
mpic++ -o hybrid_executable -fopenmp hybrid_program.cpp      ## for AOCC C++
mpic++ -o hybrid_executable -qopenmp hybrid_program.cpp      ## for Intel C++
mpic++ -o hybrid_executable -fopenmp hybrid_program.cpp      ## for GNU C++
mpic++ -o hybrid_executable -mp hybrid_program.cpp           ## for NVIDIA C++

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

When running hybrid MPI/OpenMP programs, use the MPI launcher as in MPI programs along with the $OMP_NUM_THREADS environment variable to specify the number of threads per MPI process. In the following example, four MPI processes will spawn eight threads each, for a total of 32 threads: #### run 32 hybrid threads (4 MPI procs, 8 threads per proc) export OMP_NUM_THREADS=8 mpiexec -n 4 -N 8 ./hybrid_executable [user_arguments]

Ensure the number of threads per node does not exceed the number of cores on each node. See the Slurm man page and the Batch Scheduling section for more detail on how MPI processes and threads are allocated on the nodes.

5.3. Available Compilers

Raider has four compiler suites:

• AMD Optimizing C/C++ Compiler (AOCC)
• Intel
• GNU
• NVIDIA HPC SDK

The AOCC compiler suite module is loaded by default.

Compiling can be affected by which MPI stack is being used. Raider has four MPI stacks:

• Open MPI
• Intel MPI (IMPI)
• MPICH
• MVAPICH2

For more information about MPI, or if you are using another programming model besides MPI, see Programming Models above.

All versions of MPI share a common base set of compilers that are available on both the login and compute nodes. Codes running on the login nodes must be serial. The following table lists serial compiler commands for each language.

Serial Compiler Commands

Compiler	AOCC	Intel	GNU	NVIDIA
C	clang	icc	gcc	nvc
C++	clang++	icpc	g++	nvc++
Fortran 77	flang	ifort	gfortran	nvfortran
Fortran 90	flang	ifort	gfortran	nvfortran

Codes running on compute nodes may be serial or parallel. To compile parallel codes with Open MPI, use the penguin/openmpi module and the following compiler wrappers:

Parallel Open MPI Compiler Wrapper Commands

Compiler	AOCC	Intel	GNU	NVIDIA
C	mpicc	mpicc	mpicc	mpicc
C++	mpic++	mpic++	mpic++	mpic++
Fortran 77	mpif77	mpif77	mpif77	mpif77
Fortran 90	mpif90	mpif90	mpif90	mpif90

To compile parallel codes with Intel MPI, use the intel/mpi module and the following compiler wrappers:

Parallel Intel MPI Compiler Wrapper Commands

Compiler	AOCC	Intel	GNU	NVIDIA
C	mpicc	mpicc	mpicc	mpicc
C++	mpic++	mpic++	mpic++	mpic++
Fortran 77	mpif77	mpif77	mpif77	mpif77
Fortran 90	mpif90	mpif90	mpif90	mpif90

To compile parallel codes with MPICH, use the mpich module and the following compiler wrappers:

Parallel MPICH Compiler Wrapper Commands

Compiler	AOCC	Intel	GNU	NVIDIA
C	mpicc	mpicc	mpicc	mpicc
C++	mpic++	mpic++	mpic++	mpic++
Fortran 77	mpif77	mpif77	mpif77	mpif77
Fortran 90	mpif90	mpif90	mpif90	mpif90

To compile parallel codes with MVAPICH2, use the mvapich2 module and the following compiler wrappers:

Parallel MVAPICH2 Compiler Wrapper Commands

Compiler	AOCC	Intel	GNU	NVIDIA
C	mpicc	mpicc	mpicc	mpicc
C++	mpic++	mpic++	mpic++	mpic++
Fortran 77	mpif77	mpif77	mpif77	mpif77
Fortran 90	mpif90	mpif90	mpif90	mpif90

For more information about compiling with MPI, see Programming Models above.

5.3.1. AMD Optimizing C/C++ Compiler (AOCC) Environment

The AOCC compiler system is a high performance, production quality code generation tool. The AOCC environment provides various options to users when building and optimizing C, C++, and Fortran applications. This compiler can be loaded with the amd/aocc/4.0.0 module. AOCC uses LLVM's Clang as the compiler and driver for C and C++ programs, and Flang as the compiler and driver for Fortran programs. The following table lists some of the more common options you may use:

Common AOCC Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-ffree-form	Process Fortran codes using free form
-fopenmp	Recognize OpenMP directives

Detailed information about these and other compiler options is available in the AOCC Compiler User Guide.

5.3.2. Intel Compiler Environment

The Intel compiler is a highly optimizing compiler typically producing very fast executables for Intel processors. This compiler can be loaded with the compiler/intel module. The following table lists some of the more common options you may use:

Common Intel Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-fPIC	Generate position-independent code for shared libraries
-ip	Single-file interprocedural optimization. See the sections on Compiler Optimization and Code Profiling
-ipo	Multi-file interprocedural optimization. See the sections on Compiler Optimization and Code Profiling
-free	Process Fortran codes using free form
-convert big_endian	Big-endian files; the default is little-endian
-qopenmp	Recognize OpenMP directives
-Bdynamic	Compiling using shared objects
-fpe-all=0	Trap floating point, divide by zero, and overflow exceptions

Detailed information about these and other compiler options is available in the Intel compiler (ifort, icc, and icpc) man pages.

5.3.3. GNU Compiler Collection (GCC)

The GCC Programming Environment is a popular open-source compiler typically found on all Linux systems and generally works in a compatible manner across these systems. It provides many options that are the same for all compilers in the suite. This compiler can be loaded with the compiler/gcc module. The following table lists some of the more common options you may use:

Common GCC Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-fPIC	Generate position-independent code for shared libraries
-fconvert=big=endian	Read/write big-endian files; the default is for little-endian
-Wextra -Wall	Turns on increased error reporting

Detailed information about these and other compiler options is available in the GNU compiler (gfortran, gcc, and g++) man pages.

5.3.4. NVIDIA HPC Software Development Kit (SDK)

The NVIDIA HPC SDKis a comprehensive suite of compilers and libraries enabling users to program the entire HPC platform from the GPU to the CPU and through the interconnect. The NVIDIA HPC SDK C, C++, and Fortran compilers support GPU acceleration of HPC modeling and simulation applications with standard C++ and Fortran, OpenACC directives and CUDA. This compiler can be loaded with the nvidia/nvhpc module. The following table lists some of the more common options you may use:

Common NVIDIA HPC SDK Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-fPIC	Generate position-independent code for shared libraries
-acc	Enable parallelization using OpenACC directives. By default, the compilers will parallelize and offload OpenACC regions to an NVIDIA GPU
-gpu	Control the type of GPU for which code is generated, the version of CUDA to be targeted, and several other aspects of GPU code generation
-Mfree	Compile free form Fortran
-Minfo=acc	Prints diagnostic information to STDERR regarding whether the compiler was able to produce GPU code successfully

Detailed information about these and other compiler options is available in the NVIDIA compiler (nvc, nvc++, and nvfortran) man pages.

5.4. Libraries

Several scientific and math libraries are available on Raider. The libraries provided by the vendor and/or compiler are typically faster than the open-source equivalents (CSE).

5.4.1. AMD Optimizing CPU Libraries (AOCL)

AMD Optimizing CPU Libraries (AOCL) is a set of numerical libraries optimized for the AMD EPYC processor family. The libraries can work with either the AOCC or GNU compilers.

Most users, on most codes, find better performance by using calls to AOCL routines in their applications instead of calls to public domain or user-written versions.

The AOCL collection, available in C and Fortran, contains the following scientific libraries:

• AOCL-BLIS - Basic Linear Algebra Subprograms (BLAS) - Levels 1, 2, and 3
• AOCL-Sparse - Basic linear algebra subroutines for sparse matrices and vectors
• AOCL-LibM (AMD Math Library) - Collection of basic math functions
• AOCL- libFLAME - Linear Algebra PACKage (LAPACK)
• AOCL-RNG - pseudo-random Number Generator (PRNG) Library
• AOCL-SecureRNG (Secure RNG Library)
• AOCL-Cryptography - Cryptographic functions
• AOCL-LibMem - Memory/String functions
• AOCL-Compression - Framework of lossless data compression and decompression methods
• AOCL-ScaLAPACK - Scalable LAPACK (distributed-memory parallel set of LAPACK routines)
• AOCL-FFTW - Fastest Fourier Transform in the West Routines (FFTW versions 2 and 3)

5.4.2. Intel Math Kernel Library (MKL)

Raider provides the Intel Math Kernel Library (Intel MKL), a set of numerical routines tuned specifically for Intel platform processors and optimized for math, scientific, and engineering applications. The routines, which are available via both Fortran and C interfaces, include:

• LAPACK plus BLAS (Levels 1, 2, and 3)
• ScaLAPACK plus PBLAS (Levels 1, 2, and 3)
• Fast Fourier Transform (FFT) routines for single-precision, double-precision, single-precision complex, and double-precision complex data types
• Discrete Fourier Transforms (DFTs)
• Fast Math and Fast Vector Library
• Vector Statistical Library Functions (VSL)
• Vector Transcendental Math Functions (VML)

The MKL routines are part of the Intel Programming Environment, as Intel's MKL is bundled with the Intel Compiler Suite.

Linking to the Intel Math Kernel Libraries can be complex and is beyond the scope of this introductory guide. Documentation explaining the full feature set along with instructions for linking can be found at the Intel Math Kernel Library documentation page.

Intel also makes a link advisor available to assist users with selecting proper linker and compiler options: https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html.

5.4.3. Additional Libraries

There is also an extensive set of math, I/O, and other libraries available in the $CSE_HOME directory on Raider. Information about these libraries can be found on the Baseline Configuration website at BC policy FY13-01 and the CSE Quick Reference Guide.

5.5. Debuggers

Raider has the following debugging tools: Forge DDT and the GNU Project Debugger (gdb). These debugging tools range from simple command-line debuggers to separately licensed third-party GUI tools. They can perform a variety of tasks ranging from analyzing core files to setting breakpoints and debugging running parallel programs. As a rule, your code must be compiled using the -g command-line option.

For in-depth training on using debuggers, visit the PET Knowledge Management Learning System and search for "debug" or use the following search link.

5.5.1. Forge DDT

Note: Raider is not yet accessible from the AFRL HPC Portal, and SRD is not currently available on Raider. Users will be notified as soon as these issues are resolved.

DDT supports threads, MPI, OpenMP, C/C++, Fortran, Co-Array Fortran, UPC, and CUDA. Memory debugging and data visualization are supported for large-scale parallel applications. The Parallel Stack Viewer is a unique way to see the program state of all processes and threads at a glance.

DDT is a graphical debugger; therefore, you must be able to display it via a UNIX X-Windows interface. There are several ways to do this including SSH X11 Forwarding, HPC Portal, or SRD. Follow the steps below to use DDT via X11 Forwarding or Portal.

1. Choose a remote display method: X11 Forwarding, HPC Portal, or SRD. X11 Forwarding is easier but typically very slow. HPC Portal requires no extra clients and is typically fast. SRD requires an extra client but is typically fast and may be a good option if doing a significant amount of X11 Forwarding.
   1. To use X11 Forwarding:
      1. Ensure an X server is running on your local system. Linux users will likely have this by default, but MS Windows users need to install a third-party X Windows solution. There are various options available.
      2. For Linux users, connect to Raider using ssh -Y. Windows users need to use PuTTY with X11 forwarding enabled (Connection->SSH->X11->Enable X11 forwarding).
   2. Or to use HPC Portal:
      1. Navigate to https://centers.hpc.mil/portal.
      2. Select HPC Portal at AFRL.
      3. Select XTerm | AFRL | Raider.
   3. Or, for information on using SRD, see the SRD User Guide.
2. Compile your program with the -g option.
3. Submit an interactive job, as in the following example: salloc --x11 -A Project_ID -q debug -N 1 -n 16 --ntasks-per-node=128 -t 01:00:00
4. Load the Forge DDT module: module load forge
5. Start program execution: ddt -n 4 ./my_mpi_program arg1 arg2 ... (Example for four MPI ranks)
6. The DDT window will pop up. Verify the application name and number of MPI processes. Click "Run".

An example of using DDT can be found in $SAMPLES_HOME/Programming/DDT_Example on Raider. For more information on using DDT, see the Forge User's Manual. There is also a PET course available: Debugging and Optimizing Parallel Codes with Arm Forge (MAP and DDT).

5.5.2. GNU Project Debugger (gdb)

The gdb debugger is a source-level debugger that can be invoked either with a program for execution or a running process id. It is serial-only. To launch your program under gdb for debugging, use the following command: gdb a.out corefile

To attach gdb to a program that is already executing on a node, use the following command: gdb a.out pid

For more information, the GDB manual can be found at http://www.gnu.org/software/gdb.

5.6. Code Profiling

Profiling is the process of analyzing the execution flow and characteristics of your program to identify sections of code that are likely candidates for optimization, which increases the performance of a program by modifying certain aspects for increased efficiency.

We provide several profiling tools: Forge MAP, gprof, and VTune to assist in the profiling process. In addition, a basic overview of optimization methods with information about how they may improve the performance of your code can be found in the Techniques for Improving Performance guide.

For in-depth training on using profiling tools, visit the PET Knowledge Management Learning System and search for "profiling" or use the following search link.

5.6.1. Forge MAP

The MAP profiler is a scalable, low-overhead tool to display how your program is spending its time and potentially reveal the causes of slow performance. It profiles C, C++, Fortran, and Python with no relinking, instrumentation, or code changes (though you must compile with -g). It also works with MPI (potentially at large scales), OpenMP, threads, and I/O.

To use MAP, load the Forge module: module load forge

MAP can be used interactively or in offline mode. To use it interactively, follow the interactive job instructions 1-4 from the Forge DDT Section but run the map command instead of ddt to start program execution, as follows: map -n 4 ./my_mpi_program arg1 arg2 ... (Example for four MPI ranks)

Detailed optimization information is now in Techniques for Improving Performance.

Using MAP in offline mode produces a profile (.map) output file that can be analyzed later, at your convenience, and without an actively running (potentially long, very large core-hour) job. This is more efficient for profiling at larger scales. To use MAP in offline mode, modify your batch script to include the forge module and run your application as follows: map --profile -n 4 ./my_mpi_program arg1 arg2 ... (Example for four MPI ranks)

You may view the resulting .map file in the Forge GUI. This can be done on a login node on Raider in Forge by following the interactive job instructions 1-4 from the Forge DDT Section and skipping step 3. Or you can download a free client from the Linaro Forge site, transfer the .map file, and open it on your local system. Note, the Forge client version must match the version of Forge on Raider.

For more information on using MAP, see the Forge User's Manual. There is also a PET course available: Debugging and Optimizing Parallel Codes with Arm Forge (MAP and DDT).

5.6.2. GNU Project Profiler (gprof)

The gprof profiler shows how your program is spending its time and which function calls are made. It works best for serial codes but can be used for small parallel codes, though it will not provide MPI or threaded information.

To profile code using gprof, use the -pg option during compilation. It will automatically generate profile information when executed. Use the gprof command to view the profile information. See man gprof on Raider or the gprof web site for more information.

5.6.3. Intel VTune

Intel's VTune Amplifier is a performance profiling tool for C, C++, and Fortran code that can identify where in the code time is being spent in both serial and threaded applications.

To use VTune, load the Vtune module: module load intel/vtune

To profile code using VTune, use the -g option during compilation. It will automatically generate debug information when executed. Use the vtune or vtune-gui commands to view the profile information. See man vtune on Raider or the VTune website for more information.

5.6.4. Additional Profiling Tools

There is also a set of profiling tools available in CSE. Information about these tools may be found on the Baseline Configuration website at BC policy FY13-01 and the CSE Quick Reference Guide.

5.7. Compiler Optimization Options

The -Olevel option enables code optimization when compiling. The level you choose (0-4 depending upon the compiler) determines how aggressive the optimization will be. Increasing levels of optimization may increase performance significantly but may also cause a loss of precision. There are additional options that may enable further optimizations. The following table contains the most commonly used options.

Compiler Optimization Options

Option	Purpose	Compiler Suite
-O0	No Optimization. (default in GNU)	All
-O1	Scheduling within extended basic blocks is performed. Some register allocation is performed. No global optimization	All
-O2	Level 1 plus traditional scalar optimizations such as induction recognition and loop invariant motion are performed by the global optimizer. Generally safe and beneficial. (default in Cray and Intel)	All
-O3	Levels 1 and 2 plus more aggressive code hoisting and scalar replacement optimizations that may or may not be profitable. Generally beneficial	All
-Os	Similar to the level -O2, but with extra optimizations to reduce the code size.	AOCC
-fipa-*	The GNU compilers automatically enable IPA at various -O levels. To set these manually, see the options beginning with -fipa in the gcc man page	GNU
-finline-functions	Enables function inlining within a single file	Intel
-ip	Enables interprocedural optimization within single files at a time	Intel
-ipon	Enables interprocedural optimization between files and produces up to n object files (default: n=0)	Intel
-inline-level=n	Number of levels of inlining (default: n=2)	Intel
-opt-reportn	Generate optimization report with n levels of detail	Intel
-xHost	Generate code with the highest vector instruction set available on the processor	Intel
-fp-model model	Used to tune the float-point optimizations, typically to override -On. -O3 uses model=fast which may be considered too imprecise for scientific codes, so often -O3 is used in conjunction with -fp-model precise, consistent, or strict	Intel

6. Batch Scheduling

6.1. Scheduler

The Slurm Workload Manager (Slurm) is currently running on Raider. It schedules jobs, manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request. Slurm can manage both single-processor and multiprocessor jobs. The appropriate module is automatically loaded for you when you log in. This section is merely a brief introduction to Slurm; please see the Raider Slurm Guide for more details.

6.2. Queue Information

The following table describes the Slurm queues available on Raider:

6.3. Interactive Logins

When you log in to Raider, you will be running in an interactive shell on a login node. The login nodes provide login access for Raider and support such activities as compiling, editing, and general interactive use by all users. Please note the AFRL DSRC Login Node Abuse policy. The preferred method to run resource-intensive interactive executions is to use an interactive batch session (see Interactive Batch Sessions below).

6.4. Batch Request Submission

Slurm batch jobs are submitted via the sbatch command. The format of this command is: sbatch [--directive1[=value1] --directive2[=value2]...] batch_script_file sbatch options may be specified on the command line or embedded in the batch script file by lines beginning with #SBATCH. Some of these options are discussed in Batch Resource Directives below. The batch script file is not required for interactive batch sessions (see Interactive Batch Sessions).

For a more thorough discussion of Slurm Batch Submission, see the Raider Slurm Guide.

6.5. Batch Resource Directives

Batch resource directives allow you to specify how your batch jobs should be run and the resources your job requires. Although Slurm has many directives, you only need to know a few to run most jobs.

Slurm directives can be specified in your batch script or on the command line. The syntax for a batch file is as follows: #SBATCH --directive1[=value1] #SBATCH --directive2[=value2] ...

Command lines may use sbatch or salloc depending on whether you are submitting for batch processing or running interactively. Syntax is as follows: sbatch --directive1[=value1] --directive2[=value2] ... salloc --directive1[=value1] --directive2[=value2] ...

Some directives may require values. For example, to request 32 processes per node, use the following: #SBATCH --ntasks-per-node=32

The sbatch command requires a batch file. For salloc, a batch file is optional. If no batch file is specified, then all required directives must be specified on the command line, as follows: salloc --nodes=N1 --ntasks=N2 --account=Project_ID --qos=Queue_Name --time=HH:MM:SS ...

You must specify the desired maximum walltime (HH:MM:SS), Project_ID, and Queue_Name. N1 is the number of nodes requested. N2 is the total number of tasks (usually one task per core unless specified otherwise).

Note, command-line use is required for interactive batch sessions (see Interactive Batch Sessions) since no batch file is specified.

The following directives are required for all jobs:

Required Slurm Directives

Directive (Long form)	Short form	Description
--account=Project_ID	-A	ID of the project
--qos=Queue_Name	-q	Name of the queue
--nodes=# --ntasks=# --ntasks-per-node=#	-N -n	Total number of cores (across all nodes) Total number of tasks (across all nodes) The number of cores to use per node (Note: You must use any two of these three directives, but ntasks-per-node defaults to 1, if not defined.)
--time=HH:MM:SS	-t	Maximum wall time in hours, minutes, and seconds. (Note: Additional time formats are supported; see man srun)

The following directives are optional but are commonly used:

Optional Directives

Directive (Long form)	Short form	Description
--gres=gpu:a100:#		Number of MLA nodes requested
--gres=gpu:a40:#		Number of visualization nodes requested
--gres=gpu:#		Number of non-specific GPU nodes requested
--constraint=Node_Type	-C	Specifies the specific node type requested
--job-name=Job_Name	-J	Name of the job
--error=File_Name	-e	Redirect standard error to the named file
--output=File_Name	-o	Redirect standard output to the named file
--pty		Request a shell for an interactive job
--export Variable_List		Export environment variables to the job. Use "ALL" to export all

A more complete listing of batch resource directives is available in the Raider Slurm Guide.

6.6. Interactive Batch Sessions

An interactive batch session allows you to run interactively (in a command shell) on a compute node after waiting in the batch queue.

To use the interactive batch environment, you must first acquire an interactive batch shell. This is done by using the salloc command. For example: salloc your_slurm_options --x11

The Slurm options for your job are described in Batch Resource Directives above. The command will run using your default shell. The --x11 option enables X-Windows access, so it may be omitted if your interactive job does not use a GUI.

Your interactive batch sessions will be scheduled as normal batch jobs are scheduled depending on the other queued batch jobs, so it may take some time. Once your interactive batch shell starts, you will be logged into the first compute node of those assigned to your job. At this point, you can run or debug interactive applications, execute job scripts, post-process data, etc. You can launch parallel applications on your assigned compute nodes by using an MPI or other parallel launch command.

The HPC Interactive Environment (HIE) provides an HIE queue that is specifically for interactive jobs. It offers longer job times and has nodes reserved only for HIE, so queue wait times are sometimes much shorter. However, HIE usually has limitations, such as only allowing the use of a single node at a time. See the HIE User Guide for more information.

6.7. Launch Commands

There are different commands for launching parallel executables, including MPI, from within a batch job depending on which MPI implementation or other parallel library your code uses. See the Programming Models section for more information on launching executables within a batch session.

6.8. Sample Scripts

The following example is a good starting template for a batch script to run a serial job for one hour:

#!/bin/bash
# The line above specifies the shell to use for parsing the script.
#
# Specify name of the job                   (Optional Directive)
#SBATCH --job-name=serialjob
#
# Append std output to file serialjob.out   (Optional Directive)
#SBATCH --output=serialjob.out
#
# Append std error to file serialjob.err    (Optional Directive)
#SBATCH --error=serialjob.err
#
# Specify Project ID to be charged          (Required Directive)
#SBATCH --account=Project_ID
#
# Request wall clock time of 1 hour         (Required Directive)
#SBATCH --time=01:00:00
#
# Specify queue name                        (Required Directive)
#SBATCH --qos=standard
#
# Specify the number of nodes requested     (Required Directive)
#SBATCH --nodes=1
#
# Specify the number of tasks per node      (Optional Directive)
#SBATCH --ntasks-per-node=1
#
# Change to the specified directory, in this case, the user's work directory
cd $WORKDIR
#
# Execute the serial executable on 1 core
./serial_executable
# End of batch job

The first few lines tell Slurm to save the standard output and error output to the given files and give the job a name. Skipping ahead, we estimate the run-time to be about one hour, which we know is acceptable for the standard batch queue. We need one core in total, so we request one core. The resource allocation is one full 128-core node for exclusive use by the job.

Important! Except for jobs in the transfer queue, which use shared nodes, jobs on standard nodes are charged for full 128-core nodes, even if you do not use all cores on the node.

The following example is a good starting template for a batch script to run a parallel (MPI) job for two hours:

#!/bin/bash
#
## Required Slurm Directives --------------------------------------
#SBATCH --account=Project_ID
#SBATCH --qos=standard
#SBATCH --nodes=2
# ntasks-per-node is not defined so it defaults to 1
#SBATCH --time=02:00:00
#
## Optional Slurm Directives --------------------------------------
#SBATCH --job-name=Test_Run_1
#SBATCH --export=ALL
#
## Execution Block ----------------------------------------------
# Environment Setup
# Get sequence number of unique job identifier
JOBID=`echo $SLURM_JOB_ID`
#
# create and cd to job-specific directory in your personal directory
# in the scratch file system ($WORKDIR/$JOBID)
#
mkdir $WORKDIR/$JOBID
cd $WORKDIR/$JOBID
#
# Launching
# copy executable from $HOME and execute it with a .out output file
#
cp $HOME/my_mpi_program .
mpiexec -n 256 ./my_mpi_program > my_mpi_program.out
#
# Don't forget to archive and clean up your results (see the AFRL DSRC Archive Guide for details)

We estimate the run time to be about two hours, which we know is acceptable for the standard batch queue. The optional Slurm lines give the job a name and import all environmental variables. This job is requesting two nodes, which is 256 total cores and 128 cores per node. The default value for number of cores per node is 128.

A common concern for MPI users is the need for more memory for each process. By default, one MPI process is started on each core of a node. This means on Raider standard nodes, the available memory on the node is split 128 ways. To allow an individual process to use more of the node's memory, you need to start fewer processes on each node. To do this, you must request more nodes from Slurm but run on fewer cores on each. For example, the following Slurm statements request four nodes with 128 cores per node, but it only uses 16 of those cores for MPI processes on each node:

#!/bin/bash
#### Starts 64 MPI processes; only 16 on each node
#SBATCH --nodes=4
#SBATCH --ntasks-per-node=16
#SBATCH --account=Project_ID
#SBATCH --qos=standard
#SBATCH --time=02:00:00
#
## execute on 4 nodes, total of 64 MPI processes across all
mpiexec -n 64 ./a.out
#
# Don't forget to archive and clean up your results (see the AFRL DSRC Archive Guide for details)

Further sample scripts can be found in the Raider Slurm Guide and in the Sample Code Repository ($SAMPLES_HOME) on the system. There is also an extensive discussion in the AFRL DSRC Archive Guide of sample scripts to perform data staging in the transfer queue using chained batch scripts to archive and clean up your work directory results files.

6.9. Slurm Commands

The following commands provide the basic functionality for using the Slurm batch system:

Submit jobs for batch processing: sbatch [sbatch_options] my_job_script

Check the status of submitted jobs:

scontrol show job JOBID             ##check one job
squeue -u my_user_name   ##check all of your jobs

Kill queued or running jobs: scancel JOBID

A more complete list of Slurm commands is available in the Raider Slurm Guide.

6.10. Determining Time Remaining in a Batch Job

Knowing the time remaining before the batch system will kill a job lets you write restart files or even prepare input for the next job submission. However, adding such capability to an existing source code requires knowledge to query the batch system as well as parsing the resulting output to determine the amount of remaining time.

The DoD HPCMP allocated systems now have the library, WLM_TIME, as an easy way to provide the remaining time in the batch job to C, C++, and Fortran programs. The library can be added to your job using the following:

For C: #include <wlm_time.h> void wlm_time_left(long int *seconds_left)

For C++: extern "C" { #include <wlm_time.h> } void wlm_time_left(long int *seconds_left)

For Fortran: SUBROUTINE WLM_TIME_LEFT(seconds_left) INTEGER seconds_left

For simplicity, wall-clock-time remaining is returned as an integer value of seconds.

To simplify usage, a module file defines the process environment, and a pkg-config metadata file defines the necessary compiler linker options:

For C: module load wlm_time $(CC) ctest.c `pkg-config --cflags --libs wlm_time`

For C++: module load wlm_time $(CXX) Ctest.C `pkg-config --cflags --libs wlm_time`

For Fortran: module load wlm_time $(F90) test.f90 `pkg-config --cflags-only-I --libs wlm_time`

WLM_TIME works currently with Slurm. The developers expect WLM_TIME will continue to provide a uniform interface encapsulating the underlying aspects of the workload management system.

6.11. Advance Reservations

A subset of Raider's nodes has been set aside for use as part of the Advance Reservation Service (ARS). The ARS allows users to reserve a user-designated number of nodes for a specified number of hours starting at a specific date/time. This service enables users to execute interactive or other time-critical jobs within the batch system environment. The ARS is accessible on the web at https://reservation.hpc.mil. Authentication is required. For more information, see the ARS User Guide.

7. Software Resources

7.1. Application Software

A complete list of the software versions installed on Raider can be found on the Software page. The general rule is that the two latest versions of all COTS software packages are maintained on our systems. For convenience, modules are also available for most COTS software packages. The following are other available software-related services:

• The Software License Buffer provides access to commercial software licenses on compute nodes. See the SLB User Guide.
• Singularity is the approved software for running and building containers. Containers allow you to deploy or use applications with all their software dependencies packaged together. See the Introduction to Singularity.
• The HPCMP Portal is a web interface for several graphics and web-based applications. It also includes virtual desktops for most HPC systems. See the HPC Portal Page.

  Note: Raider is not yet accessible from the AFRL HPC Portal. Users will be notified as soon as they can access the system via the Portal.

• The Secure Remote Desktop (SRD) is a client-based VNC virtual desktop application that supports graphical acceleration on GPU nodes for intensive visualization. See the SRD User Guide.
• GitLab is a web-based source code management platform. See the GitLab User Guide.

7.2. Useful Utilities

The following utilities are available on Raider. For command-line syntax and examples of usage, please see each utility's online man page.

Baseline Configuration and Other Useful Commands and Tools

Name	Description
archive	Perform basic file-handling operations on the archive system
bcmodule	An enhanced version of the standard module command
check_license	Check the status of licenses for HPCMP shared applications
cqstat	Display information about running and pending batch jobs
node_use	Display the amount of free and used memory for login nodes
qflag	Report a problem with a batch job to the HPC Help Desk
qpeek	Display spooled stdout and stderr for an executing batch job
qview	Display information about batch jobs and queues
show_queues	Report current batch queue status, usage, and limits
show_storage	Display disk/file usage and quota information
show_usage	Display CPU allocation and usage by subproject
tube	Copy files to a remote system using Kerberos host authentication

7.3. Sample Code Repository

The Sample Code Repository is a directory that contains examples for COTS batch scripts, building and using serial and parallel programs, data management, and accessing and using serial and parallel math libraries. The $SAMPLES_HOME environment variable contains the path to this area and is automatically defined in your login environment. Below is a listing of the examples provided in the Sample Code Repository on Raider.

Sample Code Repository on Raider

Applications Application-specific examples; interactive job submit scripts; use of the application name resource; software license use.
Sub-Directory	Description
abaqus	Basic batch script and input deck for an Abaqus application.
ale3d	Basic batch script and input deck for an ALE3D application.
ansys	Basic batch script and input deck for an ANSYS application.
cfd++	Basic batch script and input deck for a CFD++ application.
cfx	Basic batch script and input deck for an ANSYS CFX application.
fluent	Basic batch script and input deck for a FLUENT (now ACFD) application.
fun3d	Basic batch script for a FUN3D application.
lsdyna	Basic batch script and input deck for a LS-DYNA application.
openfoam	Basic batch script and input deck for an OPENFOAM application.
sierra	Basic batch script
starccm+	Basic batch script and input deck for a STRACCM+ application.
Data_Management Archiving and retrieving files; Lustre striping; file searching; $WORKDIR use.
MPSCP_Example	Directory containing a README file giving examples of how to use the mpscp command to transfer files between Raider and remote systems.
Postprocess_Example	Sample batch script showing how to submit a transfer queue job at the end of your computation job.
Transfer_Example	Sample batch script showing how to stage data out after a job executes using the transfer queue.
Documentation User Documentation
There are currently no samples available in this category.
Parallel_Environment MPI, OpenMP, and hybrid examples; large number of nodes jobs; single-core jobs; large memory jobs; running multiple applications within a single batch job.
Hybrid	Simple MPI/OpenMP hybrid example and batch script.
Large_Memory_Jobs	A sample large-memory jobs script.
Multiple_Jobs_per_Node	Sample Slurm job scripts for running multiple jobs on the same node.
OpenMP	A simple Open MP example and batch script.
Programming Basic code compilation; debugging; use of library files; static vs. dynamic linking; Makefiles; Endian conversion.
GPU_Examples	Several examples demonstrating use of system tools, compilation techniques, and Slurm scripts to generate and execute code using the GPGPU accelerators on Raider.
Large_Memory_Example	Simple example of how to run a job using Large-Memory nodes.
Open_Files_Limits	This example discusses the maximum number of simultaneously open files an MPI process may have, and how to adjust the appropriate settings in a Slurm job.
SO_Compile	Simple example of creating a SO (Shared Object) library and using it to compile and running against it on the compute nodes.
Timers_Fortran	Serial Timers using Fortran Intrinsics f77 and f90/95.
User_Environment Use of modules; customizing the login environment.
Module_Swap_Example	Instructions for using module swap command.
Workload_Management Basic batch scripting; use of the transfer queue; job arrays; job dependencies; Secure Remote Desktop; job monitoring.
Core_Info_Example	Sample code for generating the MPI process/core or OpenMP thread/core associativity in compute jobs.
Hybrid_Example	Simple MPI/OpenMP hybrid example and batch script.
Job_Array_Example	Instructions and example job script for using job arrays.
Job_Dependencies_Example	Example scripts on how to use Slurm job dependencies

8. Links to Vendor Documentation

8.1. Penguin Links

Penguin Home: https://www.penguinsolutions.com

8.2. RedHat Links

RedHat Home: https://www.redhat.com

8.3. AMD Links

AMD Home: https://developer.amd.com

AMD Optimizing C/C++ Compiler: https://developer.amd.com/amd-aocc

AMD AOCC User Guide: https://www.amd.com/content/dam/amd/en/documents/pdfs/developer/aocc/aocc-v4.0-ga-user-guide.pdf

8.4. Intel Links

Intel Home: https://intel.com

Intel Documentation: https://software.intel.com/en-us/intel-software-technical-documentation

Intel Compiler List: https://software.intel.com/en-us/intel-compilers

8.5. GNU Links

GNU Home: https://www.gnu.org

GNU Compiler: https://gcc.gnu.org

8.6. NVIDIA Links

NVIDIA Home: https://developer.nvidia.com

NVIDIA Documentation: https://docs.nvidia.com

NVIDIA HPC SDK: https://developer.nvidia.com/hpc-sdk

8.7. Scripting Language Links

JavaScript/Node.js: https://nodejs.org/en

Go Programming Language: https://golang.org

Perl Programming Language: https://www.perl.org

Python Programming Language: https://python.org

8.8. Debugger and Profiler Links

Forge Documentation: https://www.linaroforge.com/documentation

Intel VTune Profiler User Guide: https://www.intel.com/content/www/us/en/docs/vtune-profiler/user-guide/2023-2/overview.html

GNU Debugger Guide: https://sourceware.org/gdb/current/onlinedocs/gdb.html

9. Glossary

Batch Job

:

a single request for a set of compute nodes along with a set of tasks (usually in the form of a script) to perform on those nodes

Batch-scheduled

:

users request compute nodes via commands to batch scheduler software and wait in a queue until the requested nodes become available

Compute Node

:

a node that performs computational tasks for the user. There may be multiple types of compute nodes for specialized purposes.

Distributed Memory Model

:

a programming methodology where memory is distributed across multiple nodes giving processes on each node faster direct access to local memory, but requiring slower techniques such as message passing to access memory on other nodes

Interconnect

:

a specialized, very high-speed network that connects the nodes of an HPC system together. It is typically used for application inter-process communication (e.g., message passing) and I/O traffic.

Kerberos

:

authentication and encryption software required by the HPCMP to access HPC system login nodes and other resources. See Kerberos & Authentication

Login Node

:

a node that serves as the user's entry point into an HPC system

Node

:

an individual server in a cluster or collection of servers of an HPC system

Parallel File System

:

A software component designed to store data across multiple networked servers and to facilitate high-performance access through simultaneous, coordinated input/output operations (IOPS) between clients and storage nodes.

Shared Memory Model

:

a programming methodology where a set of processors (such as the cores within one node) have direct access to a shared pool of memory