Common Set of Environment Variables

BC Project: FY05-04
Date of Policy: 14 Jul 2005
Last Updated: 27 Sep 2019 (see Revision Log)

This policy defines a core set of environment variables that represent the same entity at each of the HPCMP allocated centers participating in the Baseline Configuration (BC) Initiative. These variables shall be predefined in the user login, batch and compute environments, making them automatically available to users at each center.

This list of environment variables is subject to change, and will be reviewed on a periodic basis for potential additions or deletions.

The current list of variables is as follows:

$ARCHIVE_HOME

This is an individual user's directory on the archival file system that serves a given compute platform. $ARCHIVE_HOME is intended to be used as a long-term file storage area by a user. It is not intended to be used by executing programs as this can significantly slow the I/O portion of a program.

$ARCHIVE_HOST

This is the hostname of the archival system serving a particular compute platform. In cases where the archival system is not network mounted to the compute platform, the combination of $ARCHIVE_HOST and $ARCHIVE_HOME will give the exact location of a user's long-term archival storage.

$BC_NODE_TYPE

This variable contains the type of node on which the variable is queried. Values of $BC_NODE_TYPE will be: LOGIN, STANDARD, PHI, BIGMEM, BATCH or ACCELERATOR. Note that BATCH is a node type used by some architectures such as Cray to launch jobs onto other nodes.

$BC_CORES_PER_NODE

This variable contains the number of CPU cores per node for the node type on which the variable is queried.

$BC_STANDARD_NODE_CORES

This variable contains the number of CPU cores per node for a standard compute node and will be set on all nodes including login nodes.

$BC_ACCELERATOR_NODE_CORES

This variable contains the number of CPU cores per node for a compute node which features CPUs and a hosted accelerator processor such as GPU or non-self-hosted Phi. The variable will be set on all nodes including login nodes if the system has any ACCELERATOR nodes. It will be set to 0 on systems without ACCELERATOR nodes.

$BC_BIGMEM_NODE_CORES

This variable contains the number of cores per node for a big memory (BIGMEM) compute node and will be set on all nodes including login nodes if the system has any BIGMEM nodes. It will be set to 0 on systems without BIGMEM nodes.

$BC_PHI_NODE_CORES

This variable contains the number of Phi cores per node, and will be set on all nodes including login nodes if the system has any Phi nodes. It will be set to 0 on systems without Phi nodes.

Example of Thunder Node Configuration
Login
Nodes
Compute Nodes
Standard
Memory
Large
Memory
GPU
Accelerated
Phi
Accelerated
Cores 28 36 36 2 x 2880 2 x 61
Example of Onyx Node Configuration
Login
Nodes
Compute Nodes
Standard
Memory
Large
Memory
GPU
Accelerated
KNL
Cores 22 44 44 1 x 3584 64
Example of Thunder and Onyx Values
Variable Thunder Onyx
$BC_STANDARD_NODE_CORES 36 44
$BC_ACCELERATOR_NODE_CORES 28 22
$BC_BIGMEM_NODE_CORES 36 44
$BC_PHI_NODE_CORES 122 64
$BC_HOST

This variable is defined as the generic (not node specific) hostname of the system in lowercase format. Examples include centennial, lightning, onyx and armstrong.

$BC_MEM_PER_NODE

This variable contains the approximate maximum memory per node available to an end user program (in integer MBytes) for the compute node type to which a job is being submitted.

$BC_MPI_TASKS_ALLOC

This variable, intended to be referenced from inside a job script, contains the number of MPI tasks that are allocated for a particular job.

$BC_NODE_ALLOC

This variable, intended to be referenced from inside a job script, contains the number of nodes allocated for a particular job.

$CC

This variable is set by any system supporting a C compiler. If a default compiler is provided, this variable is set to the default C compiler. Any subsequent change to the C compiling environment should reset this variable to the C compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.

$CENTER

This variable contains the path for a user's directory on the Center Wide File System (CWFS).

$COST_HOME

This variable contains the path to the base directory of the default installation of the Common Open Source Tools (COST) installed on a particular compute platform. (See BC policy FY13-01 for COST details.)

$CXX

This variable is set by any system supporting a C++ compiler. If a default compiler is provided, this variable is set to the default C++ compiler. Any subsequent change to the C++ compiling environment should reset this variable to the C++ compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.

$DAAC_HOME

This variable points to the top level directory for the DAAC (Data Analysis and Assessment Center) supported tools, named daac. For more information, please see BC policy FY08-01 (DAAC Supported Visualization Tools)."

$F77

This variable is set by any system supporting a Fortran 77 compiler. If a default compiler is provided, this variable is set to the default Fortran 77 compiler. Any subsequent change to the Fortran 77 compiling environment should reset this variable to the Fortran 77 compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.

$F90

This variable is set by any system supporting a Fortran 90 compiler. If a default compiler is provided, this variable is set to the default Fortran 90 compiler. Any subsequent change to the Fortran 90 compiling environment should reset this variable to the Fortran 90 compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.

$JAVA_HOME

This variable contains the path to the base directory of the default installation of JAVA on a particular compute platform.

$KRB5_HOME

This variable is set to point to the installation directory of the Kerberos utilities.

$PET_HOME

This variable contains the path to the system-wide accessible directory containing the tools formerly installed and maintained by the PET staff. Software in $PET_HOME are considered as experimental or are being evaluated for widespread use. Certain packages in $PET_HOME will be migrated to $COST_HOME, as appropriate

$PROJECTS_HOME

This variable contains the path to the base directory that contains subdirectories for user-supported applications and codes under BC policy FY10-07 (Common Location to Maintain Codes)"

$SAMPLES_HOME

This variable contains the path to the base directory for the sample codes and scripts installed by a center's staff under BC policy FY06-15 (Sample Code Repository)."

$WORKDIR

This is an individual user's directory on the local temporary file system (i.e., local high-speed disk) that is available on all HPCMP high performance computing (HPC) systems. $WORKDIR is intended to be used by executing programs to perform file I/O that is local to that system in order to avoid slower file I/O across a network mounted file system, such as a user's home or archive directories. It is not intended to be used as a permanent file storage area by users. Accordingly, this file system is NOT backed up or exported to any other system. In the event of file or directory structure deletion or a catastrophic disk failure, such files and directory structures are lost. Thus, it is the user's responsibility to transfer files that need to be saved to a location that allows for permanent file storage, such as the user's archival or home directory locations.


Revision Log
Date Revision
27 Sep 2019Changed "PETTT" to "PET"
18 Dec 2018The policy is modified to explicitly state that all environment variables shall be predefined in the user login, batch and compute environments
20 Sep 2018BC Team Audit - Deprecated $CSI_HOME, re-added $PET_HOME as an evaluation area
10 Oct 2017Revised definition of $BC_CORES_PER_NODE and introduced the following five environment variables: $BC_NODE_TYPE, $BC_STANDARD_NODE_CORES, $BC_ACCELERATOR_NODE_CORES, $BC_BIGMEM_NODE_CORES and $BC_PHI_NODE_CORES
22 May 2014Added $COST_HOME, deprecated $PET_HOME, removed SLM language from $ARCHIVE_HOME and $ARCHIVE_HOST, and stipulated lowercase for hostnames in $BC_HOST
12 Dec 2013$DAAC_HOME re-defined
31 May 2012$HOST and $KRB5HOME replaced by $BC_HOST and $KRB5_HOME respectively
16 Mar 2012BC Team Audit
02 Sep 2011$CENTER and $PROJECTS_HOME added to the list
14 Apr 2009BC_ Variables added to the list
15 Jan 2009$DAAC_HOME added to the list
10 Jul 2008$CSI_HOME and $SAMPLES_HOME added to the list