1. Introduction

1.1. Document Scope and Assumptions

This document provides an overview and introduction to the use of the HPE Cray EX4000 (Carpenter) located at the ERDC DSRC, along with a description of the specific computing environment on the system. The intent of this guide is to provide information that will enable the average user to perform computational tasks on the system. To receive the most benefit from the information provided here, you should be proficient in the following areas:

• Use of the Linux operating system
• Use of an editor (e.g., vi or emacs)
• Remote use of computer systems via network
• A selected programming language and its related tools and libraries

1.2. DSRC Policies

All policies are discussed in the Policies Section of the ERDC DSRC Introductory Site Guide. All users running at the ERDC DSRC are expected to know, understand, and follow the policies discussed. If you have any questions about the ERDC DSRC's policies, please contact the HPC Help Desk.

1.3. Obtaining an Account

To begin the account application process, visit the Obtaining an Account page and follow the instructions presented there. An HPC Help Desk video is available to guide you through the process.

1.4. Training

Training on a number of topics in this User Guide is available at the PET Knowledge Management Learning System. New account holders should strongly consider attending HPCMP New Account Orientation, which is provided via live webcast every month and available as an on-demand video.

1.5. Requesting Assistance

The HPC Help Desk is available to assist users with unclassified problems, issues, or questions. Technicians are on duty 8:00 a.m. to 8:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).

• Service Portal: https://helpdesk.hpc.mil/hpc
• E-mail: help@helpdesk.hpc.mil
• Phone: 1-877-222-2039 or 937-255-0679

You can contact the ERDC DSRC in any of the following ways:

• E-mail: dsrchelp@erdc.hpc.mil
• Phone: 1-800-500-4722 or 601-634-4400
• Fax: 601-634-5126
• U.S. Mail:
  U.S. Army Engineer Research and Development Center
  ATTN: CEERD-IH-D HPC Service Center
  3909 Halls Ferry Road
  Vicksburg, MS 39180-6199

For more information about requesting assistance, see the HPC Help Desk dropdown.

2. System Configuration

2.1. System Summary

Carpenter is an HPE Cray EX4000. It has 10 login nodes Node - an individual server in a cluster or collection of servers of an HPC system and three types of compute nodes for job execution. Carpenter uses HPE Slingshot as its high-speed interconnect Interconnect - a specialized, very high-speed network that connects the nodes of an HPC system together. It is typically used for application inter-process communication (e.g., message passing) and I/O traffic. for MPI messages and IO traffic. Carpenter uses Lustre to manage its parallel file system Parallel File System - A software component designed to store data across multiple networked servers and to facilitate high-performance access through simultaneous, coordinated input/output operations (IOPS) between clients and storage nodes..

2.2. Login and Compute Nodes

Carpenter is intended as a batch-scheduled Batch-scheduled - users request compute nodes via commands to batch scheduler software and wait in a queue until the requested nodes become available HPC system with numerous nodes. Its login nodes Login Node - a node that serves as the user's entry point into an HPC system are for minor setup, housekeeping, and job preparation tasks and are not used for large computational (e.g., memory, IO, long executions) work. All executions that require large amounts of system resources must be sent to the compute nodes Compute Node - a node that performs computational tasks for the user. There may be multiple types of compute nodes for specialized purposes. by batch job Batch Job - a single request for a set of compute nodes along with a set of tasks (usually in the form of a script) to perform on those nodes submission. Node types such as "Standard", "Large-Memory", "Visualization", etc. are considered compute nodes. Carpenter uses both shared Shared Memory Model - a programming methodology where a set of processors (such as the cores within one node) have direct access to a shared pool of memory and distributed Distributed Memory Model - a programming methodology where memory is distributed across multiple nodes giving processes on each node faster direct access to local memory, but requiring slower techniques such as message passing to access memory on other nodes memory models. Memory is shared among all the cores on one node but is not shared among the nodes across the cluster.

Carpenter's login nodes use AMD 9654 Genoa processors with 8 GB of usable memory. All memory and cores on the node are shared among all users who are logged in. Therefore, users should not use more than 8 GB of memory at any one time.

Carpenter's standard compute nodes use AMD 9654 Genoa processors. Each node contains 349 GB of usable shared memory. Standard compute nodes are intended for standard compute tasks.

Carpenter's large-memory compute nodes use AMD 9654 Genoa processors. Each node contains 2.973 TB of usable shared memory. Each also contains 8.8 TB of on-node NVMe SSD storage. Large-memory compute nodes are intended for tasks requiring additional memory.

Carpenter's visualization nodes consist of an AMD 7713 Milan processor paired with an NVIDIA A40 PCIe 4 GPU. Each node contains 467 GB of usable shared memory on the node, as well as 48 GB of shared memory internal to each accelerator. Visualization nodes are intended for tasks requiring a GPU.

3. Accessing the System

3.1. Kerberos

For security purposes, you must have a current Kerberos Kerberos - authentication and encryption software required by the HPCMP to access HPC system login nodes and other resources. See Kerberos & Authentication ticket on your computer before attempting to connect to Carpenter. To obtain a ticket you must either install a Kerberos client kit on your desktop or connect via the HPC Portal. Visit the Kerberos & Authentication page for information about installing Kerberos clients on your Windows, Linux, or Mac desktop. Instructions are also available on those pages for getting a ticket and logging into the HPC systems from each platform.

3.2. Logging In

The system host name for the Carpenter cluster is carpenter.erdc.hpc.mil, which redirects you to one of 10 login nodes. Hostnames and IP addresses to these nodes are available upon request from the HPC Help Desk.

The preferred way to login to Carpenter is via ssh, as follows: % ssh username@carpenter.erdc.hpc.mil

3.3. File Transfers

File transfers to DSRC systems (except for those to the local archive system) must be performed using the following HPCMP Kerberized tools: scp and sftp. Windows users may use a graphical secure file transfer protocol (sftp) client such as FileZilla. See the HPC Help Desk Video on Using FileZilla. Before using any of these tools, you must use a Kerberos client to obtain a Kerberos ticket. Information about installing and using a Kerberos client can be found on the Kerberos & Authentication page.

The command below uses secure copy (scp) to copy a single local file into a destination directory on a Carpenter login node. % scp local_file username@carpenter.erdc.hpc.mil:/target_dir

The scp command can be used to send multiple files. This command transfers all files with the .txt extension to the same destination directory. % scp *.txt username@carpenter.erdc.hpc.mil:/target_dir

The example below uses the secure file transfer protocol (sftp) to connect to Carpenter, then uses sftp's cd and put commands to change to the destination directory and copy a local file there. The sftp quit command ends the sftp session. Use the sftp help command to see a list of all sftp commands. % sftp username@carpenter.erdc.hpc.mil sftp> cd target_dir sftp> put local_file sftp> quit

4. User Environment

4.1. User Directories

The following user directories are provided for all users on Carpenter:

File Systems on Carpenter

Path	Formatted Capacity	File System Type	Storage Type	User Quota	Minimum File Retention
/p/home ($HOME)	919 TB	Lustre	HDD	100 GB	None
/p/work ($WORKDIR)	3.8 PB	Lustre	NVMe SSD	50 TB	21 Days
/p/cwfs ($CENTER)	3.3 PB	GPFS	HDD	100 TB	120 Days
/p/global ($GLOBAL)	25 PB	Lustre	SSD/HDD	None	60 Days
/p/app ($PROJECTS_HOME)	460 TB	Lustre	HDD	None	None

4.1.1. Home Directory ($HOME)

When you log in, you are placed in your home directory, /p/home/username. It is accessible from the login and compute nodes and can be referenced by the environment variable $HOME.

Your home directory is intended for storage of frequently used files, scripts, and small utility programs. It has a 100-GB quota, and files stored there are not subject to automatic deletion based on age. It is backed up weekly to enable file restoration in the event of catastrophic system failure.

Important! The home file system is not tuned for parallel I/O and does not support application-level I/O. Jobs performing intensive file I/O in your home directory will perform poorly and cause problems for everyone on the system. Running jobs should use the work file system ($WORKDIR) or global shared file system ($GLOBAL)for file I/O.

4.1.2. Work Directory ($WORKDIR)

The work file system is a large, high-performance Lustre-based file system tuned for parallel application-level I/O. It is accessible from the login and compute nodes and provides temporary file storage for queued and running jobs.

All users have a work directory, /p/work/username, on this file system, which can be referenced by the environment variable, $WORKDIR. This directory should be used for application file I/O. NEVER allow your jobs to perform file I/O in $HOME.

$WORKDIR has a quota of 50 TB. It is not backed up or exported to any other system and is subject to an automated deletion cycle. If available disk space gets too low, files that have not been accessed in 21 days may be deleted. If this happens or if catastrophic disk failure occurs, lost files are irretrievable. To prevent the loss of important files, transfer them to a long-term storage area, such as your archival directory ($ARCHIVE_HOME, see Archive Usage), which has no quota. Or, for smaller files, your home directory ($HOME).

Maintaining the high performance and stability of the Lustre file system is important for the efficient and effective use of Carpenter by all users. For example, setting stripe counts can maximize your performance and prevent you from filling up a single file system component causing system instability. Additional examples can be found in $SAMPLES_HOME/Data_Management/OST_Stripes on Carpenter.

To avoid errors that can arise from two jobs using the same scratch directory, a common technique is to create a unique subdirectory for each batch job. See Sample Scripts for an example of a script that does this.

4.1.3. Center Directory ($CENTER)

The Center-Wide File System (CWFS) is an NFS-mounted file system. It is accessible from the login nodes of all HPC systems at the center and from the HPC Portal. It provides centralized, shared storage that enables users to easily access data from multiple systems. The CWFS is not tuned for parallel I/O and does not support application-level I/O.

All users have a directory on the CWFS. The name of your directory may vary between systems and between centers, but the environment variable $CENTER always refers to this directory.

$CENTER has a quota of 100 TB. It is not backed up or exported to any other system and is subject to an automated deletion cycle. If available disk space gets too low, files that have not been accessed in 120 days may be deleted. If this happens or if catastrophic disk failure occurs, lost files are irretrievable. To prevent the loss of important files, transfer them to a long-term storage area, such as your archival directory ($ARCHIVE_HOME, see Archive Usage), which has no quota. Or, for smaller files, your home directory ($HOME).

4.1.4. Global Directory ($GLOBAL)

The global file system is a very large, high-performance Lustre-based file system tuned for parallel application-level I/O. It is accessible from the login and compute nodes and provides temporary file storage for queued and running jobs.

All users have a global directory, /p/global/username, on this file system, which can be referenced by the environment variable, $GLOBAL. This directory should be used for application file I/O. NEVER allow your jobs to perform file I/O in $HOME.

$GLOBAL does not have a quota limit. It is not backed up or exported to any other system and is subject to an automated deletion cycle. If available disk space gets too low, files that have not been accessed in 60 days may be deleted. If this happens or if catastrophic disk failure occurs, lost files are irretrievable. To prevent the loss of important files, transfer them to a long-term storage area, such as your archival directory ($ARCHIVE_HOME, see Archive Usage), which has no quota. Or, for smaller files, your home directory ($HOME).

Maintaining the high performance and stability of the Lustre file system is important for the efficient and effective use of Carpenter by all users. For example, setting stripe counts can maximize your performance and prevent you from filling up a single file system component causing system instability. You are expected to take steps to ensure your file storage and access methods follow the suggested guidelines in the ERDC DSRC Lustre Guide. Additional examples can be found in $SAMPLES_HOME/Data_Management/OST_Stripes on Carpenter.

To avoid errors that can arise from two jobs using the same scratch directory, a common technique is to create a unique subdirectory for each batch job. See Sample Scripts for an example of a script that does this.

4.1.5. Projects Directory ($PROJECTS_HOME)

The Projects directory, $PROJECTS_HOME, is a file system set aside for group-shared storage. It is intended for storage of semi-permanent files, similar to a home directory, but typically larger and shared by a group. It is not meant for high-speed application output ($WORKDIR, see Work Directory). A new project sub-directory can be created via an HPC Help Desk request and appears as follows: $PROJECTS_HOME/new_group_dir. The HPC Help Desk request must specify a UNIX group to be assigned to the project sub-directory. Users can create and manage UNIX groups in the Portal to the Information Environment, allowing the creator of the assigned group to manage the members of the group with access to the project sub-directory.

4.1.6. Storage On-node ($LOCALWORKDIR)

Some compute nodes (see the large-memory nodes in the Node Configuration Table) include a local solid-state storage device (NVMe SSD) that is local to and accessible by the node only and can be accessed by the environment variable $LOCALWORKDIR. It has improved local bandwidth and latency, but each device is a separate drive with no parallel read/write capability. Files stored on this device must be relocated at the end of a job or they are lost when the node is reassigned to a new job.

4.1.7. Specialized Temporary Directories

Each node includes several specialized directories.

The /tmp and /var/tmp directories are usually intended for temporary files as created by the operating system. Do not use these directories for your own files, as filling up these file systems can cause issues.

Carpenter also provides a "virtual" file system (i.e., "RAM disk") called /dev/shm which is local to each compute node. You may use this file system to store files in memory. It automatically increases in size as needed, up to half of the memory of the node. It is extremely fast, but it is also small and takes available node memory away from your application. An example use case is performing significant I/O with many small files when the memory is not otherwise needed by the application.

4.2. Shells

The following shells are available on Carpenter: csh, bash, ksh, tcsh, sh, and zsh.

To change your default shell, log into the Portal to the Information Environment and go to "User Information Environment" > "View/Modify personal account information". Scroll down to "Preferred Shell" and select your desired default shell. Then scroll to the bottom and click "Save Changes". Your requested change should take effect within 24 hours.

4.3. Environment Variables

A number of environment variables are provided by default on all HPCMP high performance computing (HPC) systems. We encourage you to use these variables in your scripts where possible. Doing so will help simplify your scripts and reduce portability issues if you ever need to run those scripts on other systems.

4.3.1. Common Environment Variables

The following environment variables are automatically set in both your login and batch environments:

Common Environment Variables

Variable	Description
$ARCHIVE_HOME	Your directory on the archive system
$ARCHIVE_HOST	The host name of the archive system
$BC_ACCELERATOR_NODE_CORES	The number of CPU cores per node for a compute node which features CPUs and a hosted accelerator processor
$BC_BIGMEM_NODE_CORES	The number of cores per node for a big memory (BIGMEM) compute node
$BC_CORES_PER_NODE	The number of CPU cores per node for the node type on which the variable is queried
$BC_HOST	The generic (not node specific) name of the system. Examples include mustang, carpenter and gaffney
$BC_NODE_TYPE	The type of node on which the variable is queried. Values of $BC_NODE_TYPE are: LOGIN, STANDARD, PHI, BIGMEM, BATCH, or ACCELERATOR
$BC_STANDARD_NODE_CORES	The number of CPU cores per node for a standard compute node
$CC	The currently selected C compiler. This variable is automatically updated when a new compiler environment is loaded
$CENTER	Your directory on the Center-Wide File System (CWFS)
$CSE_HOME	The top-level directory for the Computational Science Environment (CSE) tools and applications
$CXX	The currently selected C++ compiler. This variable is automatically updated when a new compiler environment is loaded
$DAAC_HOME	The top level directory for the DAAC (Data Analysis and Assessment Center) supported tools
$F77	The currently selected Fortran 77 compiler. This variable is automatically updated when a new compiler environment is loaded
$F90	The currently selected Fortran 90 compiler. This variable is automatically updated when a new compiler environment is loaded
$GLOBAL	Your temporary directory on the global shared file system
$HOME	Your home directory on the system
$JAVA_HOME	The directory containing the default installation of JAVA
$KRB5_HOME	The directory containing the Kerberos utilities
$LOCALWORKDIR	A high-speed work directory that is local and unique to an individual node, if the node provides such space
$PET_HOME	The directory containing tools installed by PET staff, which are considered experimental or under evaluation. Certain older packages have been migrated to $CSE_HOME, as appropriate
$PROJECTS_ARCHIVE	The directory on the archive system in which user-supported applications, code, and data may be kept
$PROJECTS_HOME	The directory in which user-supported applications and codes may be installed
$SAMPLES_HOME	A directory that contains the Sample Code Repository, a variety of sample codes and scripts provided by a center's staff
$WORKDIR	Your work directory on the local temporary file system (i.e., local high-speed disk)

4.3.2. Batch-Only Environment Variables

In addition to the variables listed above, the following variables are automatically set only in your batch environment. That is, your batch scripts can see them when they run. These variables are supplied for your convenience and are intended for use inside your batch scripts.

Batch-Only Environment Variables

Variable	Description
$BC_MEM_PER_NODE	The approximate maximum memory (in integer MB) per node available to an end user program for the compute node type to which a job is being submitted
$BC_MPI_TASKS_ALLOC	The number of MPI tasks allocated for a particular job
$BC_NODE_ALLOC	The number of nodes allocated for a particular job

Please refer to the Carpenter PBS Guide for a number of helpful environment variables provided during batch runs.

4.4. Archive Usage

All our HPC systems have access to an online archival mass storage system that provides long-term storage for users' files on a petascale tape file system that resides on a robotic tape library system. A 2.6-PB disk cache frontends the tape file system and temporarily holds files while they are being transferred to or from tape.

Tape file systems have very slow access times. The tapes must be robotically pulled from the tape library, mounted in one of the limited number of tape drives, and wound into position for file archival or retrieval. Individual files under 2GB are not desirable for archive. For this reason, users should always tar up their small files in a large tarball when archiving a significant number of files. A good size range for tarballs is about 2 TB - 4 TB. At that size, the time required for file transfer and tape I/O is reasonable. Files larger than 4 TB will greatly increase the time required for both archival and retrieval.

The environment variable $ARCHIVE_HOME is automatically set for you and can be used to reference your archive directory when using archive commands.

4.4.1. Archive Command Synopsis

A synopsis of the archive utility is listed below. For information on additional capabilities, see the ERDC DSRC Archive Guide or read the online man page available on each system. The archive command is non-Kerberized and can be used in batch submission scripts if desired.

Copy one or more files from the archive system: archive get [-C path] [-s] file1 [file2...]

List files and directory contents on the archive system: archive ls [lsopts] [file/dir ...]

Create directories on the archive system: archive mkdir [-C path] [-m mode] [-p] [-s] dir1 [dir2 ...]

Copy one or more files to the archive system: archive put [-C path] [-s] file1 [file2 ...]

Move or rename files and directories on the archive server: archive mv [-C path] [-s] file1 [file2 ...] target

Remove files and directories from the archive server: archive rm [-C path] [-r] [-s] file1 [file2 ...]

Check and report the status of the archive server: archive stat [-s]

Remove empty directories from the archive server: archive rmdir [-C path] [-p] [-s] dir1 [dir2 ...]

Change permissions of files and directories on the archive server: archive chmod [-C path] [-R] [-s] mode file1 [file2 ...]

Change the group of files and directories on the archive server: archive chgrp [-C path] [-R] [-h] [-s] group file1 [file2 ...]

5. Program Development

5.1. Modules

Software modules are a convenient way to set needed environment variables and include necessary directories in your path so commands for particular applications can be found. Carpenter also uses modules to initialize your environment with application software, system commands, libraries, and compiler suites.

A number of modules are loaded automatically as soon as you log in. To see the currently loaded modules, use the module list command. To see the entire list of available modules, use the module avail command. You can modify the configuration of your environment by loading and unloading modules. For complete information on how to do this and other information on using modules, see the ERDC DSRC Modules Guide.

5.2. Programming Models

Carpenter supports several parallel programming models. A programming model augments a programming language with parallel processing capability. Different programming models may use a different approach to express parallelism, such as message passing, threads, distributed memory, shared memory, etc.

Note, if an application is not programmed for distributed memory, then only the cores on a single node can be used. This is limited to 192 cores on Carpenter's standard nodes. See the Node Configuration table for core counts on other nodes.

Note, keep the system architecture in mind during code development. For instance, if your program requires more memory than is available on a single node, then you need to parallelize your code so it can function across multiple nodes.

Key supported programming models are discussed in each subsection below.

5.2.1. Message Passing Interface (MPI)

Carpenter's default MPI stack supports the MPI 3.1 Standard. MPI is part of the software support for parallel programming across a network of computer systems through a technique known as message passing. MPI establishes a practical, portable, efficient, and flexible standard for high-performance message passing. See man intro_mpi for additional information.

When creating an MPI program, ensure the default MPI module (cray-mpich) or other available MPI module (mpi, openmpi) is loaded. To check this, run the module list command. To load the desired module, run the following command: module load cray-mpich

Also, ensure the source code contains one of the following for the MPI library:

INCLUDE "mpif.h"        ## for older Fortran
USE mpi                 ## for newer Fortran
#include <mpi.h>        ## for C/C++

To compile an MPI program, use one of the following:

ftn -o MPI_executable mpi_program.f       ## for Fortran
cc -o MPI_executable mpi_program.c        ## for C
CC -o MPI_executable mpi_program.cpp      ## for C++

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

To run an MPI program within a batch script, load the same modules as used to compile the application before using the following command to launch your executable: mpiexec -n mpi_procs ./MPI_executable [user_arguments] where mpi_procs is the number of MPI processes being started. For example: #### The following starts 192 MPI processes #### (the placement of the processes on nodes is handled by the batch scheduler) mpiexec -n 192 ./MPI_executable

For more information on which MPI Standard features are supported by the default MPI on the system, check the BC MPI Test Suite page.

5.2.2. Open Multi-Processing (OpenMP)

OpenMP is a portable, scalable model that gives programmers a simple and flexible interface for developing parallel applications. It supports shared-memory multiprocessing programming in C, C++, and Fortran and consists of a set of compiler directives, library routines, and environment variables that influence compilation and run-time behavior.

When creating an OpenMP program, if using OpenMP functions (e.g., omp_get_wtime), ensure the source code includes one of the following lines:

INCLUDE "omp.h"        ## for older Fortran
USE omp_lib            ## for newer Fortran
#include <omp.h>       ## for C/C++

To compile an OpenMP program, use the following compiler commands and flags:

ftn -o OpenMP_executable openmp_program.f             ## for Cray Fortran
ftn -o OpenMP_executable -qopenmp openmp_program.f    ## for Intel Fortran
ftn -o OpenMP_executable -fopenmp openmp_program.f    ## for GNU Fortran
ftn -o OpenMP_executable -fopenmp openmp_program.f    ## for AOCC Fortran
ftn -o OpenMP_executable -mp openmp_program.f         ## for NVIDIA Fortran

cc -o OpenMP_executable openmp_program.c	         ## for Cray C
cc -o OpenMP_executable -qopenmp openmp_program.c     ## for Intel C
cc -o OpenMP_executable -fopenmp openmp_program.c     ## for GNU C
cc -o OpenMP_executable -fopenmp openmp_program.c     ## for AOCC C
cc -o OpenMP_executable -mp openmp_program.c          ## for NVIDIA C

CC -o OpenMP_executable openmp_program.cpp            ## for Cray C++
CC -o OpenMP_executable -qopenmp openmp_program.cpp   ## for Intel C++
CC -o OpenMP_executable -fopenmp openmp_program.cpp   ## for GNU C++
CC -o OpenMP_executable -fopenmp openmp_program.cpp   ## for AOCC C++
CC -o OpenMP_executable -mp openmp_program.cpp        ## for NVIDIA C++

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

When running OpenMP applications, the $OMP_NUM_THREADS environment variable must be used to specify the number of threads. For example: #### run 32 threads on one node export OMP_NUM_THREADS=32 ./OpenMP_executable [user_arguments]

In the example above, the application starts the OpenMP_executable on one node and spawns a total of 32 threads. Since Carpenter has 192 cores per compute node, if you wanted to run one thread per core, you would set $OMP_NUM_THREADS to 192 instead.

5.2.3. Hybrid MPI/OpenMP

An application built with the hybrid model of parallel programming can run using both OpenMP and Message Passing Interface (MPI). This allows the application to run on multiple nodes yet leverages OpenMP's advantages within each node. In hybrid applications, multiple OpenMP threads are spawned by MPI processes, but MPI calls should not be issued from OpenMP parallel regions or by an OpenMP thread.

When creating a hybrid MPI/OpenMP program, follow the instructions in both the MPI and OpenMP sections above for creating your program.

To compile a hybrid program, use the MPI compilers in conjunction with the OpenMP options, as follows:

ftn -o hybrid_executable  hybrid_program.f            ## for Cray Fortran
ftn -o hybrid_executable -qopenmp hybrid_program.f    ## for Intel Fortran
ftn -o hybrid_executable -fopenmp hybrid_program.f    ## for GNU Fortran
ftn -o hybrid_executable -fopenmp hybrid_program.f    ## for AOCC Fortran
ftn -o hybrid_executable -mp hybrid_program.f         ## for NVIDIA Fortran

cc -o hybrid_executable  hybrid_program.c              ## for Cray C
cc -o hybrid_executable -qopenmp hybrid_program.c      ## for Intel C
cc -o hybrid_executable -fopenmp hybrid_program.c      ## for GNU C
cc -o hybrid_executable -fopenmp hybrid_program.c      ## for AOCC C
cc -o hybrid_executable -mp hybrid_program.c           ## for NVIDIA C

CC -o hybrid_executable  hybrid_program.cpp  ## for Cray C++
CC -o hybrid_executable -qopenmp hybrid_program.cpp  ## for Intel C++
CC -o hybrid_executable -fopenmp hybrid_program.cpp  ## for GNU C++
CC -o hybrid_executable -fopenmp hybrid_program.cpp  ## for AOCC C++
CC -o hybrid_executable -mp hybrid_program.cpp       ## for NVIDIA C++

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

When running hybrid MPI/OpenMP programs, use the MPI launcher as in MPI programs along with the $OMP_NUM_THREADS environment variable to specify the number of threads per MPI process. In the following example, four MPI processes will spawn eight threads each for a total of 32 threads: #### run 32 hybrid threads (4 MPI procs, 8 threads per proc) export OMP_NUM_THREADS=8 mpiexec -n 4 ./hybrid_executable [user_arguments]

Ensure the number of threads per node does not exceed the number of cores on each node. See the Batch Scheduling section for more detail on how MPI processes and threads are allocated on the nodes.

5.2.4. SHMEM

Logically shared, distributed-memory access (SHMEM) routines provide high-performance, high-bandwidth communication for use in highly parallelized scalable programs. The SHMEM data-passing library routines are similar to the MPI library routines; they pass data between cooperating parallel processes. The SHMEM data-passing routines can be used in programs that perform computations in separate address spaces and explicitly pass data to and from different processes in the program.

Cray-OpenSHMEMX works with any compiler. Just load the following modules: module load cray-mpi module load cray-dsmml module load cray-openshmemx See the intro_shmem man page for more information.

When creating a SHMEM program, load the desired compiler module and load the SHMEM modules as above. Ensure the source code includes one of the following lines:

INCLUDE "shmem.fh"       ## for Fortran
#include <shmem.h>       ## for C/C++

To compile a SHMEM program, use the following compiler wrappers:

ftn -o SHMEM_executable -shmem_option shmem_program.f     ## for Fortran
cc -o SHMEM_executable -shmem_option shmem_program.c      ## for C
CC -o SHMEM_executable -shmem_option shmem_program.cpp    ## for C++

The compiler wrappers resolve all SHMEM routine calls automatically. Specific mention of the SHMEM library is not required on the compilation line.

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers.

To run SHMEM applications, use the following command: aprun -n num_procs ./SHMEM_executable [user_arguments] The -n num_procs option indicates the number of processes to start, each process using one core.

5.2.5. Co-Array Fortran (CAF)

The Cray compiler supports Co-Array Fortran (CAF). This is a set of Partitioned Global Address Space (PGAS) extensions that lets you reference memory locations on any node without the need for message-passing protocols. This can greatly simplify writing and debugging parallel code.

To compile a CAF program, ensure the PrgEnv-cray module is loaded and use the following compiler:

ftn -o CAF_executable -h caf caf_program.f     ## for Cray compiler

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers. Other compilers not listed here may support CAF but may not be integrated to run across multiple nodes.

To run CAF applications, use the following command: aprun -n num_procs ./CAF_executable [user_arguments] The -n num_procs option indicates the number of processes to start, each process using one core.

Many users of PGAS extensions also use MPI or SHMEM calls in their codes. In such cases, be sure to use the appropriate include statements in your source code, as described in the respective sections above.

5.2.6. Unified Parallel C (UPC)

The Cray compiler supports Unified Parallel C (UPC). This is a set of Partitioned Global Address Space (PGAS) extensions that lets you reference memory locations on any node, without the need for message-passing protocols. This can greatly simplify writing and debugging a parallel code.

To compile a UPC program, ensure the PrgEnv-cray module is loaded and use the following compiler:

cc -o UPC_executable -h upc upc_program.c     ## for Cray compiler

For more information on compilers, compiler wrappers, and compiler options, see Available Compilers. Other compilers not listed here may support UPC but may not be integrated to run across multiple nodes.

To run UPC applications, use the following command: aprun -n num_procs ./UPC_executable [user_arguments] The -n num_procs option indicates the number of processes to start, each process using one core.

Many users of PGAS extensions also use MPI or SHMEM calls in their codes. In such cases, be sure to use the appropriate include statements in your source code, as described in the corresponding sections above.

5.3. Available Compilers

Carpenter has five compiler suites:

• Cray Compiler
• Intel Compiler
• GNU Compiler
• AMD Compiler
• NVIDIA Compiler

The Cray compiler suite module is loaded by default.

Compiling can be affected by which MPI stack is being used. Carpenter has three MPI stacks:

• Cray MPICH
• Intel MPI
• OpenMPI

For more information about MPI, or if you are using another programming model besides MPI, see Programming Models above.

All versions of MPI share a common base set of compilers that are available on both the login and compute nodes. Codes running on the login nodes must be serial. The following table lists serial compiler commands for each language.

Serial Compiler Commands

Compiler	Cray	Intel	GNU	AMD	NVIDIA
C	craycc	icc	gcc	clang	nvc
C++	crayCC	icpc	g++	clang++	nvc++
Fortran 77	crayftn	ifort	gfortran	flang	nvfortran
Fortran 90	crayftn	ifort	gfortran	flang	nvfortran

Codes running on compute nodes may be serial or parallel. To compile parallel codes with Cray MPICH, unload the current PrgEnv-* module, load the desired PrgEnv-compiler (see next table) module, the cray-mpich module, and the following compiler wrappers:

Parallel Cray MPICH Compiler Wrapper Commands

Compiler	Cray	Intel	GNU	AMD	NVIDIA
module load	PrgEnv-cray	PrgEnv-intel	PrgEnv-gnu	PrgEnv-aocc	PrgEnv-nvhpc
C	cc	cc	cc	cc	cc
C++	CC	CC	CC	CC	CC
Fortran 77	ftn	ftn	ftn	ftn	ftn
Fortran 90	ftn	ftn	ftn	ftn	ftn

To compile parallel codes with Intel MPI, purge all modules then load intel (or intel-classic), mpi, and cray-pals modules. Use the following compiler wrappers (only works with Intel or GNU):

Parallel Intel MPI Compiler Wrapper Commands

Compiler	Intel	GNU
C	mpiicc	mpiicc
C++	mpiicpc	mpiicpc
Fortran 77	mpiifort	mpiifort
Fortran 90	mpiifort	mpiifort

To compile parallel codes with OpenMPI, use the openmpi module and the following compiler wrappers (only works with Intel or GNU):

Parallel OpenMPI Compiler Wrapper Commands

Compiler	Intel	GNU
C	mpicc	mpicc
C++	mpicxx	mpicxx
Fortran 77	mpif77	mpif77
Fortran 90	mpif90	mpif90

For more information about compiling with MPI, see Programming Models above.

5.3.1. Cray Compiler Environment

The HPE Cray Programming environment has C, C++, and Fortran compilers that are designed to extract increased performance from the systems, regardless of the underlying architecture. This compiler can be loaded with the PrgEnv-cray module (which will load the cce module). The following table lists some of the more common options you may use:

Common Cray Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-fPIC	Generate position-independent code for shared libraries
-f free	Process Fortran codes using free form
-m0	Reports detailed information about code optimizations to stdout as compile proceeds
-f openmp	Recognize OpenMP directives (C/C++)
-homp	Recognize OpenMP directives (Fortran)
-hdynamic	Compiling using shared objects
-K traps=fp	Trap floating point, divide by zero, and overflow exceptions

Detailed information about these and other compiler options is available in the Cray compiler (craycc, crayCC, crayftn) man pages.

5.3.2. Intel Compiler Environment

The Intel compiler is a highly optimizing compiler typically producing very fast executables for Intel processors. This compiler can be loaded with the PrgEnv-intel module (which will load the intel module). The following table lists some of the more common options you may use:

Common Intel Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-fPIC	Generate position-independent code for shared libraries
-ip	Single-file interprocedural optimization. See the sections on Compiler Optimization and Code Profiling
-ipo	Multi-file interprocedural optimization. See the sections on Compiler Optimization and Code Profiling
-free	Process Fortran codes using free form
-convert big_endian	Big-endian files; the default is little-endian
-qopenmp	Recognize OpenMP directives
-Bdynamic	Compiling using shared objects
-fpe-all=0	Trap floating point, divide by zero, and overflow exceptions

Detailed information about these and other compiler options is available in the Intel compiler (ifort, icc, and icpc) man pages.

5.3.3. GNU Compiler Collection (GCC)

The GCC Programming Environment is a popular open-source compiler typically found on all Linux systems and generally works in a compatible manner across these systems. It provides many options that are the same for all compilers in the suite. This compiler can be loaded with the PrgEnv-gnu module (which will load the gcc module). The following table lists some of the more common options you may use:

Common GCC Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-fPIC	Generate position-independent code for shared libraries
-fconvert=big=endian	Read/write big-endian files; the default is for little-endian
-Wextra -Wall	Turns on increased error reporting

Detailed information about these and other compiler options is available in the GNU compiler (gfortran, gcc, and g++) man pages.

5.3.4. AOCC Compiler Environment

The AMD Optimizing C/C++ Compiler (AOCC) compiler system is a high performance, production quality code generation tool. The AOCC environment provides various options to users when building and optimizing C, C++, and Fortran applications. AOCC uses LLVM's Clang as the compiler and driver for C and C++ programs, and Flang as the compiler and driver for Fortran programs. This compiler can be loaded with the PrgEnv-aocc module (which will load the aocc module).

The following table lists some of the more common options you may use:

Common AOCC Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-ffree-form	Compile free form Fortran

Detailed information about these and other compiler options is available in the AOCC compiler (clang, clang++, and flang) man pages.

5.3.5. NVIDIA Compiler Environment

The NVIDIA HPC Software Development Kit (SDK) is a comprehensive suite of compilers and libraries enabling users to program the entire HPC platform from the GPU to the CPU and through the interconnect. The NVIDIA HPC SDK C, C++, and Fortran compilers support GPU acceleration of HPC modeling and simulation applications with standard C++ and Fortran, OpenACC directives and CUDA. This compiler can be loaded with the PrgEnv-nvhpc module (which will load the nvhpc module). The following table lists some of the more common options you may use:

Common NVIDIA Compiler Options

Option	Purpose
-c	Generate intermediate object file but do not attempt to link
-I directory	Search in directory for include or module files
-L directory	Search in directory for libraries
-o outfile	Name executable "outfile" rather than the default "a.out"
-Olevel	Set the optimization level. For more information on optimization, see the sections on Compiler Optimization and Code Profiling
-g	Generate symbolic debug information
-acc	Enable parallelization using OpenACC directives. By default, the compilers will parallelize and offload OpenACC regions to an NVIDIA GPU
-gpu	Control the type of GPU for which code is generated, the version of CUDA to be targeted, and several other aspects of GPU code generation
-Minfo=acc	Prints diagnostic information to STDERR regarding whether the compiler was able to produce GPU code successfully

Detailed information about these and other compiler options is available in the NVIDIA compiler (nvc, nvc++, and nvfortran) man pages.

5.4. Libraries

Several scientific and math libraries are available on Carpenter. The libraries provided by the vendor and/or compiler are typically faster than the open-source equivalents (CSE).

5.4.1. Cray Scientific and Math Libraries (CSML) LibSci

The CSML, also known as LibSci, is a collection of numerical routines optimized for best performance on Cray systems. All programming environment modules load cray-libsci by default, except when noted.

Most users, on most codes, find better performance by using calls to Cray LibSci routines in their applications instead of calls to public domain or user-written versions.

Note, additionally, Cray EX systems make use of the Cray LibSci Accelerator routines for enhanced performance on GPU-equipped compute nodes. For more information, see the intro_libsci_acc man page.

The LibSci collection, available in C and Fortran, contains the following scientific libraries:

• Basic Linear Algebra Subroutines (BLAS) - Levels 1, 2, and 3
• C interface to the legacy BLAS (CBLAS)
• Basic Linear Algebra Communication Subprograms (BLACS)
• Linear Algebra Package (LAPACK)
• Scalable LAPACK (ScaLAPACK) (distributed-memory parallel set of LAPACK routines)
• Fast Fourier Transform (FFT)
• Fastest Fourier Transform in the West Routines (FFTW versions 2 and 3)
• Accelerated BLAS and LAPACK routines (LibSci_ACC)

Two libraries unique to Cray are also included:

• Iterative Refinement Toolkit (IRT)
• CrayBLAS (library of BLAS routines autotuned for Cray EX series)

The IRT routines may be used by setting the environment variable $IRT_USE_SOLVERS to 1 or by coding an explicit call to an IRT routine. Additional information is available by using the man intro_irt command.

Cray modules, such as cray-libsci, are included automatically when used with the cc, CC, or ftn compiler wrappers. However, to link to the Cray LibSci libraries manually, add the entry -lsci_compiler[_mpi][_mp] where compiler can be cray, intel, gnu, nvidia, or aocc, and, optionally, _mpi and _mp select MPI-enabled and multithreaded versions, respectively.

5.4.2. Intel Math Kernel Library (MKL)

Carpenter provides the Intel MKL, a set of numerical routines tuned specifically for Intel platform processors and optimized for math, scientific, and engineering applications. The routines, which are available via both Fortran and C interfaces, include:

• LAPACK plus BLAS (Levels 1, 2, and 3)
• ScaLAPACK plus PBLAS (Levels 1, 2, and 3)
• Fast Fourier Transform (FFT) routines for single-precision, double-precision, single-precision complex, and double-precision complex data types
• Discrete Fourier Transforms (DFTs)
• Fast Math and Fast Vector Library
• Vector Statistical Library Functions (VSL)
• Vector Transcendental Math Functions (VML)

The MKL routines are part of the Intel Programming Environment as Intel's MKL is bundled with the Intel Compiler Suite.

Linking to the Intel Math Kernel Libraries can be complex and is beyond the scope of this introductory guide. Documentation explaining the full feature set along with instructions for linking can be found at the Intel Math Kernel Library documentation page.

Intel also makes a link advisor available to assist users with selecting proper linker and compiler options: https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html.

5.4.3. Other Cray-supplied Libraries

The following Cray-optimized libraries are also available:

• FFTW - Discrete Fourier Transform libraries
• HDF5 - Hierarchical Data Format library (serial and parallel)
• NETCDF - Network Common Data Format library (serial and parallel)

The modulefiles for these libraries are of the form cray-library_name. Use module avail cray- to find the appropriate modulefile. More information about linking these libraries is in the documentation, which is available after loading the associated modulefiles.

5.4.4. Additional Libraries

There is also an extensive set of math, I/O, and other libraries available in the $CSE_HOME directory on Carpenter. Information about these libraries can be found on the Baseline Configuration website at BC policy FY13-01 and the CSE Quick Reference Guide.

5.5. Debuggers

Carpenter has the following debugging tools: Cray Debugger Support Tools (CDST), Forge DDT, and GNU Project Debugger (gdb). These debugging tools range from simple command-line debuggers to separately licensed third-party GUI tools. They can perform a variety of tasks ranging from analyzing core files to setting breakpoints and debugging running parallel programs. As a rule, your code must be compiled using the -g command-line option.

For in-depth training on using debuggers, visit the PET Knowledge Management Learning System and search for "debug" or use the following search link.

5.5.1. Forge DDT

DDT supports threads, MPI, OpenMP, C/C++, Fortran, Co-Array Fortran, UPC, and CUDA. Memory debugging and data visualization are supported for large-scale parallel applications. The Parallel Stack Viewer is a unique way to see the program state of all processes and threads at a glance.

DDT is a graphical debugger; therefore, you must be able to display it via a UNIX X-Windows interface. There are several ways to do this including SSH X11 Forwarding, HPC Portal, or SRD. Follow the steps below to use DDT via X11 Forwarding or Portal.

1. Choose a remote display method: X11 Forwarding, HPC Portal, or SRD. X11 Forwarding is easier but typically very slow. HPC Portal requires no extra clients and is typically fast. SRD requires an extra client but is typically fast and may be a good option if doing a significant amount of X11 Forwarding.
   1. To use X11 Forwarding:
      1. Ensure an X server is running on your local system. Linux users will likely have this by default, but MS Windows users need to install a third-party X Windows solution. There are various options available.
      2. For Linux users, connect to Carpenter using ssh -Y. Windows users need to use PuTTY with X11 forwarding enabled (Connection->SSH->X11->Enable X11 forwarding).
   2. Or to use HPC Portal:
      1. Navigate to https://centers.hpc.mil/portal.
      2. Select HPC Portal at ERDC.
      3. Select XTerm | ERDC | Carpenter.
   3. Or, for information on using SRD, see the SRD User Guide.
2. Compile your program with the -g option.
3. Submit an interactive job, as in the following example: qsub -l select=1:ncpus=192:mpiprocs=192 -A Project_ID -l walltime=00:30:00 -q debug -X -I
4. Load the Forge DDT module: module load forge
5. Start program execution: ddt -n 4 ./my_mpi_program arg1 arg2 ... (Example for four MPI ranks)
6. The DDT window will pop up. Verify the application name and number of MPI processes. Click "Run".

An example of using DDT can be found in $SAMPLES_HOME/Programming/DDT_Example on Carpenter. For more information on using DDT, see the Forge User's Manual. There is also a PET course available: Debugging and Optimizing Parallel Codes with Arm Forge (MAP and DDT).

5.5.2. GNU Project Debugger (gdb)

The gdb debugger is a source-level debugger that can be invoked either with a program for execution or a running process id. It is serial-only. To launch your program under gdb for debugging, use the following command: gdb a.out corefile

To attach gdb to a program that is already executing on a node, use the following command: gdb a.out pid

For more information, the GDB manual can be found at http://www.gnu.org/software/gdb.

5.5.3. Cray Debugger Support Tools (CDST)

Cray provides a collection of debugging packages that include the following:

Gdb4hpc
gdb4hpc is a GDB-based parallel debugger used to debug applications compiled with CCE, Intel, and GNU C, C++, and Fortran compilers. It allows users to either launch an application or attach to an already-running application. This debugger can be accessed by loading the gdb4hpc module. Detailed information about this debugger can be found in the gdb4hpc man page on Carpenter.

Valgrind4hpc
Valgrind4pc is a Valgrind-based debugging tool used to detect memory leaks and errors in parallel applications. Valgrind4hpc aggregates any duplicate messages across ranks to help provide an understandable picture of program behavior. This tool can be accessed by loading the valgrind4hpc module. Detailed information can be found in the valgrind4hpc man page on Carpenter.

Stack Trace Analysis Tool (STAT)
STAT is a single merged stack backtrace tool to analyze application behavior at the function level. It helps trace down the cause of crashes. This tool can be accessed by loading the cray-stat module. Detailed information can be found in the STAT man page on Carpenter.

Abnormal Termination Processing (ATP)
ATP is a scalable core file generation and analysis tool for determining what causes a code to crash. It can be accessed by loading the atp module. Detailed information can be found in the atp man page on Carpenter.

Cray Comparative Debugger (CCDB)
CCDB is Cray's next generation debugging tool. It features a GUI interface that extends the comparative debugging capabilities of gdb, enabling users to easily compare data structures between two executing applications. This tool can be accessed by loading the cray-ccdb module. Detailed information can be found in the ccdb man page on Carpenter.

5.6. Code Profiling

Profiling is the process of analyzing the execution flow and characteristics of your program to identify sections of code that are likely candidates for optimization, which increases the performance of a program by modifying certain aspects for increased efficiency.

We provide two profiling tools: Forge MAP and gprof to assist in the profiling process. In addition, a basic overview of optimization methods with information about how they may improve the performance of your code can be found in the Techniques for Improving Performance guide.

For in-depth training on using profiling tools, visit the PET Knowledge Management Learning System and search for "optimiz" or use the following search link.

5.6.1. Forge MAP

The MAP profiler is a scalable, low-overhead tool to display how your program is spending its time and potentially reveal the causes of slow performance. It profiles C, C++, Fortran, and Python with no relinking, instrumentation, or code changes (though you must compile with -g). It also works with MPI (potentially at large scales), OpenMP, threads, and I/O.

To use MAP, load the Forge module: module load forge

MAP can be used interactively or in offline mode. To use it interactively, follow the interactive job instructions 1-4 from the Forge DDT Section but run the map command instead of ddt. Detailed optimization information is now in Techniques for Improving Performance module, and start program execution as follows: map -n 4 ./my_mpi_program arg1 arg2 ... (Example for four MPI ranks)

Using MAP in offline mode produces a profile (.map) output file that can be analyzed later, at your convenience, and without an actively running (potentially long, very large core-hour) job. This is more efficient for profiling at larger scales. To use MAP in offline mode, modify your batch script to include the forge module and run your application as follows: map --profile -n 4 ./my_mpi_program arg1 arg2 ... (Example for four MPI ranks)

You may view the resulting .map file in the Forge GUI. This can be done on a login node on Carpenter in Forge by following the interactive job instructions 1-4 from the Forge DDT Section and skipping step 3. Or you can download a free client from the Linaro Forge site, transfer the .map file, and open it on your local system. Note, the Forge client version must match the version of Forge on Carpenter.

For more information on using MAP, see the Forge User's Manual. There is also a PET course available: Debugging and Optimizing Parallel Codes with Arm Forge (MAP and DDT).

5.6.2. GNU Project Profiler (gprof)

The gprof profiler shows how your program is spending its time and which function calls are made. It works best for serial codes but can be used for small parallel codes, though it will not provide MPI or threaded information.

To profile code using gprof, use the -pg option during compilation. It will automatically generate profile information when executed. Use the gprof command to view the profile information. See man gprof on Carpenter or the gprof web site for more information.

5.6.3. Additional Profiling Tools

There is also a set of profiling tools available in CSE. Information about these tools may be found on the Baseline Configuration website at BC policy FY13-01 and the CSE Quick Reference Guide.

5.7. Compiler Optimization Options

The -Olevel option enables code optimization when compiling. The level you choose (0-4 depending upon the compiler) determines how aggressive the optimization will be. Increasing levels of optimization may increase performance significantly but may also cause a loss of precision. There are additional options that may enable further optimizations. The following table contains the most commonly used options.

Compiler Optimization Options

Option	Purpose	Compiler Suite
-O0	No Optimization. (default in GNU)	All
-O1	Scheduling within extended basic blocks is performed. Some register allocation is performed. No global optimization	All
-O2	Level 1 plus traditional scalar optimizations such as induction recognition and loop invariant motion are performed by the global optimizer. Generally safe and beneficial. (default in Cray and Intel)	All
-O3	Levels 1 and 2 plus more aggressive code hoisting and scalar replacement optimizations that may or may not be profitable. Generally beneficial	All
-Ofast	Levels 1-3 plus very aggressive optimizations that may violate compliance with language standards	All
-fipa-*	The GNU compilers automatically enable IPA at various -O levels. To set these manually, see the options beginning with -fipa in the gcc man page	GNU
-finline-functions	Enables function inlining within a single file	Intel
-ip	Enables interprocedural optimization within single files at a time	Intel
-ipon	Enables interprocedural optimization between files and produces up to n object files (default: n=0)	Intel
-inline-level=n	Number of levels of inlining (default: n=2)	Intel
-opt-reportn	Generate optimization report with n levels of detail	Intel
-xHost	Generate code with the highest vector instruction set available on the processor	Intel
-fp-model model	Used to tune the float-point optimizations, typically to override -On. -O3 uses model=fast which may be considered too imprecise for scientific codes, so often -O3 is used in conjunction with -fp-model precise, consistent, or strict	Intel
-flto	Enable Link Time Optimization	Cray, AOCC, NVIDIA

6. Batch Scheduling

6.1. Scheduler

The Portable Batch System (PBS) is currently running on Carpenter. It schedules jobs, manages resources and job queues, and can be accessed through the interactive batch environment or by submitting a batch request. PBS can manage both single-processor and multiprocessor jobs. The appropriate module is automatically loaded for you when you log in. This section is merely a brief introduction to PBS; please see the Carpenter PBS Guide for more details.

6.2. Queue Information

The following table describes the PBS queues available on Carpenter. Jobs with high, frontier, and standard priority are handled differently depending on the requested walltime and core count.

Users should submit directly to high, frontier, or standard queues, which are routing queues. Jobs will be moved automatically into the appropriate large job "_lg", small job "_sm", or long walltime "_lw" queues.

Job priority starts at an initial value based on core count and the queue to which the job was submitted. It then increases for each hour that the job has been waiting to run.

* The running job limit on the debug queue per user is 2.
** The running job limit on the background queue per user is 3.

6.3. Interactive Logins

When you log in to Carpenter, you will be running in an interactive shell on a login node. The login nodes provide login access for Carpenter and support such activities as compiling, editing, and general interactive use by all users. Please note the ERDC DSRC Login Node Abuse policy. The preferred method to run resource-intensive interactive executions is to use an interactive batch session (see Interactive Batch Sessions below).

6.4. Batch Request Submission

PBS batch jobs are submitted via the qsub command. The format of this command is: qsub [ options ] batch_script_file qsub options may be specified on the command line or embedded in the batch script file by lines beginning with #PBS. Some of these options are discussed in Batch Resource Directives below. The batch script file is not required for interactive batch sessions (see Interactive Batch Sessions).

For a more thorough discussion of PBS Batch Submission, see the Carpenter PBS Guide.

6.5. Batch Resource Directives

Batch resource directives allow you to specify how your batch jobs should be run and the resources your job requires. Although PBS has many directives, you only need to know a few to run most jobs.

PBS directives can be specified in your batch script or on the command line. The syntax for a batch file is as follows: #PBS -directive1 [option1[=value1]] #PBS -directive2 [option2[=value2]] ...

The syntax for the command line is as follows: qsub -directive1 [option1[=value1]] -directive2 [option2[=value2]] ...

Some options may require values. For example, to start a 32-process job, request one node of 192 cores and specify 32 processes per node, as follows: #PBS -l select=1:ncpus=192:mpiprocs=32

All required directives must be specified in the batch script or on the command line, as follows: qsub -l select=N1:ncpus=N2:mpiprocs=N3[:Nodetype] -A Project_ID -q Queue_Name -l walltime=HHH:MM:SS batch_script

You must specify the desired maximum walltime (HHH:MM:SS), Project_ID, Queue_Name, and number of nodes requested (N1). The number of cores per node (N2) for each node type is given in the node configuration table in System Summary. The number of processes per node (N3) can range from 1 to N2 and up to 2*N2 if you desire hyperthreading.

Note, command-line use is required for interactive batch sessions (see Interactive Batch Sessions) since no batch file is specified.

The Nodetype parameter is optional. To specify the node type on which your job will run, select the associated directive option from the following table:

For example, to request two large-memory nodes: #PBS -l select=2:ncpus=192:mpiprocs=192:bigmem=1

The following directives are required for all jobs:

Required PBS Directives

Directive	Description
-A Project_ID	Name of the project
-q Queue_Name	Name of the queue
-l walltime=HHH:MM:SS	Maximum wall time in hours, minutes, and seconds
-l select=#:ncpus=#:mpiprocs=#	Select sets the number of requested nodes. ncpus specifies the number of cores available on the node. mpiprocs specifies the number of MPI processes per node.

The following directives are optional but are commonly used:

Optional Directives

Directive	Description
-l select=#:ncpus=#:mpiprocs=#[:Nodetype]	A variant of the required "select" directive above. "Nodetype" is optional. Nodetype specifies the type of node (see Node Type Directives above)
-N Job_Name	Name of the job
-e File_Name	Redirect standard error to the named file
-o File_Name	Redirect standard output to the named file
-j oe	Merge standard error and standard output into standard output
-l application=Application_Name	Identify the application being used. See $SAMPLES_HOME/Application_Name/application_names on Carpenter
-I	Request an interactive batch shell
-X	Enable X-Windows for graphical applications
-V	Export all environment variables to the job
-v Variable_List	Export specific environment variables to the job

A more complete listing of batch resource directives is available in the Carpenter PBS Guide.

6.6. Interactive Batch Sessions

An interactive batch session allows you to run interactively (in a command shell) on a compute node after waiting in the batch queue.

To use the interactive batch environment, you must first acquire an interactive batch shell. This is done by adding the -I option to your qsub command. For example: qsub your_pbs_options -I

The PBS options for your job are described in Batch Resource Directives above.

Your interactive batch sessions will be scheduled as normal batch jobs are scheduled depending on the other queued batch jobs, so it may take some time. Once your interactive batch shell starts, you will be logged into the first compute node of those assigned to your job. At this point, you can run or debug interactive applications, execute job scripts, post-process data, etc. You can launch parallel applications on your assigned compute nodes by using an MPI or other parallel launch command.

The HPC Interactive Environment (HIE) provides an HIE queue that is specifically for interactive jobs. It offers longer job times and has nodes reserved only for HIE, so queue wait times are sometimes much shorter. However, HIE usually has limitations, such as only allowing the use of a single node at a time. See the HIE User Guide for more information.

6.7. Launch Commands

There are different commands for launching parallel executables, including MPI, from within a batch job depending on which MPI implementation or other parallel library your code uses. See the Programming Models section for more information on launching executables within a batch session.

6.8. Sample Scripts

The following example is a good starting template for a batch script to run a serial job for one hour:

#!/bin/bash
# The line above specifies the shell to use for parsing the script.
#
# Specify name of the job                   (Optional Directive)
#PBS -N serialjob
#
# Append std output to file serialjob.out   (Optional Directive)
#PBS -o serialjob.out
#
# Append std error to file serialjob.err    (Optional Directive)
#PBS -e serialjob.err
#
# Specify Project ID to be charged          (Required Directive)
#PBS -A Project_ID
#
# Request wall clock time of 1 hour         (Required Directive)
#PBS -l walltime=01:00:00
#
# Specify queue name                        (Required Directive)
#PBS -q standard
#
# Specify the number of cores               (Required Directive)
#PBS -l select=1:ncpus=192:mpiprocs=1
#
# Change to the specified directory, in this case, the user's work directory
cd $WORKDIR
# To change into the directory where qsub was issued, use "cd $PBS_O_WORKDIR" instead
#
# Execute the serial executable on 1 core
./serial_executable
# End of batch job

The first few lines tell PBS to save the standard output and error output to the given files and give the job a name. Skipping ahead, we estimate the run-time to be about one hour, which we know is acceptable for the standard batch queue. We need one core in total, so we request one core. The resource allocation is one full 192-core node for exclusive use by the job.

Important! Except for jobs in the transfer queue, which use shared nodes, jobs on standard nodes are charged for full 192-core nodes, even if you do not use all cores on the node.

The following example is a good starting template for a batch script to run a parallel (MPI) job for two hours:

#!/bin/bash
#
## Required PBS Directives --------------------------------------
#PBS -A Project_ID
#PBS -q standard
#PBS -l select=2:ncpus=192:mpiprocs=192
#PBS -l walltime=02:00:00
#
## Optional PBS Directives --------------------------------------
#PBS -N Test_Run_1
#PBS -j oe
#PBS -V
#
## Execution Block ----------------------------------------------
# Environment Setup
# Get sequence number of unique job identifier
JOBID=`echo $PBS_JOBID | cut -d '.' -f 1`
# create and cd to job-specific directory in your personal directory
# in the scratch file system ($WORKDIR/$JOBID)
mkdir $WORKDIR/$JOBID
cd $WORKDIR/$JOBID
#
# Launching
# copy executable from $HOME and execute it with a .out output file
cp $HOME/my_mpi_program .
mpiexec -n 384 ./my_mpi_program > my_mpi_program.out
#
# Don't forget to archive and clean up your results (see the ERDC DSRC Archive Guide for details)

We estimate the run time to be about two hours, which we know is acceptable for the standard batch queue. The optional PBS lines tell PBS to combine the standard output and error output, give the job a name, and import all environmental variables. This job is requesting 384 total cores and 192 cores per node allowing the job to run on two nodes.

A common concern for MPI users is the need for more memory for each process. By default, one MPI process is started on each core of a node. This means on Carpenter standard nodes, the available memory on the node is split 192 ways. To allow an individual process to use more of the node's memory, you need to start fewer processes on each node. To do this, you must request more nodes from PBS but run on fewer cores on each. For example, the following select statement requests four nodes with 192 cores per node, but it only uses 16 of those cores for MPI processes on each node:

#!/bin/bash
#
#### Starts 64 MPI processes; only 16 on each node
#PBS -l select=4:ncpus=192:mpiprocs=16
#PBS -A Project_ID
#PBS -q standard
#PBS -l walltime=02:00:00
#
## execute on 4 nodes, total of 64 MPI processes across all - 16 on each node.
mpiexec -n 64 ./a.out
#
# Don't forget to archive and clean up your results (see the ERDC DSRC Archive Guide for details)

Further sample scripts can be found in the Carpenter PBS Guide and in the Sample Code Repository ($SAMPLES_HOME) on the system. There is also an extensive discussion in the ERDC DSRC Archive Guide of sample scripts to perform data staging in the transfer queue using chained batch scripts to archive and clean up your work directory results files.

6.9. PBS Commands

The following commands provide the basic functionality for using the PBS batch system:

Submit jobs for batch processing: qsub [qsub_options] my_job_script

Check the status of submitted jobs:

qstat JOBID             ##check one job
qstat -u my_user_name   ##check all of your jobs

Kill queued or running jobs: qdel JOBID

A more complete list of PBS commands is available in the Carpenter PBS Guide.

6.10. Determining Time Remaining in a Batch Job

Knowing the time remaining before the batch system will kill a job lets you write restart files or even prepare input for the next job submission. However, adding such capability to an existing source code requires knowledge to query the batch system as well as parsing the resulting output to determine the amount of remaining time.

The DoD HPCMP allocated systems now have the library, WLM_TIME, as an easy way to provide the remaining time in the batch job to C, C++, and Fortran programs. The library can be added to your job using the following:

For C: #include <wlm_time.h> void wlm_time_left(long int *seconds_left)

For C++: extern "C" { #include <wlm_time.h> } void wlm_time_left(long int *seconds_left)

For Fortran: SUBROUTINE WLM_TIME_LEFT(seconds_left) INTEGER seconds_left

For simplicity, wall-clock-time remaining is returned as an integer value of seconds.

To simplify usage, a module file defines the process environment, and a pkg-config metadata file defines the necessary compiler linker options:

For C: module load wlm_time $(CC) ctest.c `pkg-config --cflags --libs wlm_time`

For C++: module load wlm_time $(CXX) Ctest.C `pkg-config --cflags --libs wlm_time`

For Fortran: module load wlm_time $(F90) test.f90 `pkg-config --cflags-only-I --libs wlm_time`

WLM_TIME works currently with PBS. The developers expect WLM_TIME will continue to provide a uniform interface encapsulating the underlying aspects of the workload management system.

6.11. Advance Reservations

A subset of Carpenter's nodes has been set aside for use as part of the Advance Reservation Service (ARS). The ARS allows users to reserve a user-designated number of nodes for a specified number of hours starting at a specific date/time. This service enables users to execute interactive or other time-critical jobs within the batch system environment. The ARS is accessible on the web at https://reservation.hpc.mil. Authentication is required. For more information, see the ARS User Guide.

7. Software Resources

7.1. Application Software

A complete list of the software installed on Carpenter can be found on the Software page. The general rule is that the two latest versions of all COTS software packages are maintained on our systems. For convenience, modules are also available for most COTS software packages. The following are other available software-related services:

• The Software License Buffer provides access to commercial software licenses on compute nodes. See the SLB User Guide.
• Singularity is the approved software for running and building containers. Apptainer is the Linux Foundation fork of the Singularity codebase that is installed on Carpenter. Containers allow you to deploy or use applications with all their software dependencies packaged together. See the Introduction to Singularity.
• The HPCMP Portal is a web interface for several graphics and web-based applications. It also includes virtual desktops for most HPC systems. See the HPC Portal Page.
• The Secure Remote Desktop (SRD) is a client-based VNC virtual desktop application that supports graphical acceleration on GPU nodes for intensive visualization. See the SRD User Guide.
• GitLab is a web-based source code management platform. See the GitLab User Guide.

7.2. Useful Utilities

The following utilities are available on Carpenter. For command-line syntax and examples of usage, please see each utility's online man page.

Baseline Configuration and Other Useful Commands and Tools

Name	Description
archive	Perform basic file-handling operations on the archive system
bcmodule	An enhanced version of the standard module command
check_license	Check the status of licenses for HPCMP shared applications
cqstat	Display information about running and pending batch jobs
node_use	Display the amount of free and used memory for login nodes
qflag	Report a problem with a batch job to the HPC Help Desk
qpeek	Display spooled stdout and stderr for an executing batch job
qview	Display information about batch jobs and queues
show_queues	Report current batch queue status, usage, and limits
show_storage	Display disk/file usage and quota information
show_usage	Display CPU allocation and usage by subproject

7.3. Sample Code Repository

The Sample Code Repository is a directory that contains examples for COTS batch scripts, building and using serial and parallel programs, data management, and accessing and using serial and parallel math libraries. The $SAMPLES_HOME environment variable contains the path to this area and is automatically defined in your login environment. Below is a listing of the examples provided in the Sample Code Repository on Carpenter.

Sample Code Repository on Carpenter

Application_Name Use of the application name resource.
Sub-Directory	Description
application_names	README and list of valid strings for application names intended for use in every PBS script preamble. The HPCMP encourages applications not specifically named in the list to be denoted as "other".
Applications Application-specific examples; interactive job submit scripts; software license use.
There are currently no samples available in this category.
Data_Management Archiving and retrieving files; Lustre striping; file searching; $WORKDIR use.
There are currently no samples available in this category.
Parallel_Environment MPI, OpenMP, and hybrid examples; large memory jobs; packing nodes with single-core jobs; running multiple applications within a single PBS job.
Hello_World_Example	Example "Hello World" codes demonstrating how to compile and execute MPI, OpenMP, and hybrid MPI/OpenMP codes using PBS on the Broadwell compute nodes. Each paradigm is contained in the corresponding named subdirectory: MPI, OPENMP, HYBRID.
Parallel_IO Tools for performing parallel I/O.
There are currently no samples available in this category.
Programming Basic code compilation; debugging; use of library files; static vs. dynamic linking; Makefiles; Endian conversion.
There are currently no samples available in this category.
Software_Containers Basic examples of how to use container technology to create a custom environment for your applications.
There are currently no samples available in this category.
User_Environment Use of modules; customizing the login environment; use of common environment variables to facilitate portability of work between systems
There are currently no samples available in this category.
Workload_Management Basic batch scripting; use of the transfer queue; job arrays; job dependencies; Secure Remote Desktop; job monitoring; generating core/MPI process or core/MPI process-OpenMP thread associativity.
BatchScript_Example	Simple PBS batch script showing all required preamble statements and a few optional statements. More advanced batch script showing more optional statements and a few ways to set up PBS jobs. Description of the system hardware.
Interactive_Example	C and Fortran code samples and scripts for running an interactive MPI job on Carpenter.
Job_Array_Example	Sample job script and qsub option for using job arrays.
Job_Dependencies_Example	Example code, scripts, and instructions demonstrating how to set up a job dependency for jobs depending on how one or more other jobs execute, or to perform some action that one or more other jobs require before execution.

8. Links to Vendor Documentation

8.1. HPE Cray Links

HPE Home: https://www.hpe.com

8.2. SUSE Links

SUSE Home: https://suse.com

SUSE Linux Enterprise Server: https://suse.com/products/server

8.3. GNU Links

GNU Home: https://www.gnu.org

GNU Compiler: https://gcc.gnu.org/onlinedocs

8.4. Intel Links

Intel Home: https://www.intel.com

Intel Documentation: https://software.intel.com/en-us/intel-software-technical-documentation

Intel Compiler List: https://software.intel.com/en-us/intel-compilers

8.5. Debugger Links

Forge Documentation: https://www.linaroforge.com/documentation

9. Glossary

Batch Job

:

a single request for a set of compute nodes along with a set of tasks (usually in the form of a script) to perform on those nodes

Batch-scheduled

:

users request compute nodes via commands to batch scheduler software and wait in a queue until the requested nodes become available

Compute Node

:

a node that performs computational tasks for the user. There may be multiple types of compute nodes for specialized purposes.

Distributed Memory Model

:

a programming methodology where memory is distributed across multiple nodes giving processes on each node faster direct access to local memory, but requiring slower techniques such as message passing to access memory on other nodes

Interconnect

:

a specialized, very high-speed network that connects the nodes of an HPC system together. It is typically used for application inter-process communication (e.g., message passing) and I/O traffic.

Kerberos

:

authentication and encryption software required by the HPCMP to access HPC system login nodes and other resources. See Kerberos & Authentication

Login Node

:

a node that serves as the user's entry point into an HPC system

Node

:

an individual server in a cluster or collection of servers of an HPC system

Parallel File System

:

A software component designed to store data across multiple networked servers and to facilitate high-performance access through simultaneous, coordinated input/output operations (IOPS) between clients and storage nodes.

Shared Memory Model

:

a programming methodology where a set of processors (such as the cores within one node) have direct access to a shared pool of memory