Common Set of Environment Variables
BC Project: FY05-04
Date of Policy: 14 Jul 2005
Last Updated: 03 Apr 2023 (see Revision Log)
This policy defines a core set of environment variables that represent the same entity at each of the HPCMP allocated centers participating in the Baseline Configuration (BC) Initiative. These variables shall be predefined in the user login, batch and compute environments, making them automatically available to users at each center.
This list of environment variables is subject to change, and will be reviewed on a periodic basis for potential additions or deletions.
The current list of variables is as follows:
$ARCHIVE_HOME
This is an individual user's directory on the archival file system that serves a
given compute platform. $ARCHIVE_HOME is intended to be used as a long-term file
storage area by a user. It is not intended to be used by executing programs as this
can significantly slow the I/O portion of a program.
$ARCHIVE_HOST
This is the hostname of the archival system serving a particular compute platform. In
cases where the archival system is not network mounted to the compute platform, the
combination of $ARCHIVE_HOST and $ARCHIVE_HOME will give the exact
location of a user's long-term archival storage.
$BC_NODE_TYPE
This variable contains the type of node on which the variable is queried. Values of
$BC_NODE_TYPE will be: LOGIN, STANDARD, PHI, BIGMEM, BATCH or ACCELERATOR.
Note that BATCH is a node type used by some architectures such as Cray to launch jobs onto
other nodes.
$BC_CORES_PER_NODE
This variable contains the number of CPU cores per node for the node type on which the variable is queried.
$BC_STANDARD_NODE_CORES
This variable contains the number of CPU cores per node for a standard compute node and will be set on all nodes including login nodes.
$BC_ACCELERATOR_NODE_CORES
This variable contains the number of CPU cores per node for a compute node which features CPUs and a hosted accelerator processor such as GPU or another non-self-hosted device. The variable will be set on all nodes including login nodes if the system has any ACCELERATOR nodes. It will be set to 0 on systems without ACCELERATOR nodes.
$BC_BIGMEM_NODE_CORES
This variable contains the number of cores per node for a big memory (BIGMEM) compute node and will be set on all nodes including login nodes if the system has any BIGMEM nodes. It will be set to 0 on systems without BIGMEM nodes.
$BC_PHI_NODE_CORES
This variable contains the number of Phi cores per node, and will be set on all nodes including login nodes if the system has any Phi nodes. It will be set to 0 on systems without Phi nodes.
| Login Nodes |
Compute Nodes | ||||
|---|---|---|---|---|---|
| Standard Memory |
Large Memory |
GPU Accelerated |
Phi Accelerated |
||
| Cores | 48 | 48 | 48 | 1 x 3584 | 0 |
| Login Nodes |
Compute Nodes | ||||
|---|---|---|---|---|---|
| Standard Memory |
Large Memory |
GPU Accelerated |
KNL | ||
| Cores | 22 | 44 | 44 | 1 x 3584 | 64 |
| Variable | Koehr | Onyx |
|---|---|---|
| $BC_STANDARD_NODE_CORES | 48 | 44 |
| $BC_ACCELERATOR_NODE_CORES | 48 | 22 |
| $BC_BIGMEM_NODE_CORES | 48 | 44 |
| $BC_PHI_NODE_CORES | 0 | 64 |
$BC_HOST
This variable is defined as the generic (not node specific) hostname of the system in lowercase format. Examples include centennial, mustang, onyx and koehr.
$BC_MEM_PER_NODE
This variable contains the approximate maximum memory per node available to an end user program (in integer MB) for the compute node type to which a job is being submitted.
$BC_MPI_TASKS_ALLOC
This variable, intended to be referenced from inside a job script, contains the number of MPI tasks that are allocated for a particular job.
$BC_NODE_ALLOC
This variable, intended to be referenced from inside a job script, contains the number of nodes allocated for a particular job.
$CC
This variable is set by any system supporting a C compiler. If a default compiler is provided, this variable is set to the default C compiler. Any subsequent change to the C compiling environment should reset this variable to the C compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.
$CENTER
This variable contains the path for a user's directory on the Center Wide File System (CWFS).
$COST_HOME
This variable contains the path to the base directory of the default installation of the Common Open Source Tools (COST) installed on a particular compute platform. COST has been deprecated in favor of the Computational Science Environment (CSE). Starting with TI-19 systems, $COST_HOME is not loaded.
$CSE_HOME
This variable contains the path to the base directory of the default installation of the Computational Science Environment (CSE) installed on a particular computer platform. (See BC policy FY13-01 for CSE details.)
$CXX
This variable is set by any system supporting a C++ compiler. If a default compiler is provided, this variable is set to the default C++ compiler. Any subsequent change to the C++ compiling environment should reset this variable to the C++ compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.
$DAAC_HOME
This variable points to the top level directory for the DAAC (Data Analysis and Assessment Center) supported tools, named daac. For more information, please see BC policy FY08-01 (DAAC Supported Visualization Tools)."
$F77
This variable is set by any system supporting a Fortran 77 compiler. If a default compiler is provided, this variable is set to the default Fortran 77 compiler. Any subsequent change to the Fortran 77 compiling environment should reset this variable to the Fortran 77 compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.
$F90
This variable is set by any system supporting a Fortran 90 compiler. If a default compiler is provided, this variable is set to the default Fortran 90 compiler. Any subsequent change to the Fortran 90 compiling environment should reset this variable to the Fortran 90 compiler for that environment. The variable refers to the compiler required for program execution on the system's compute nodes.
$JAVA_HOME
This variable contains the path to the base directory of the default installation of JAVA on a particular compute platform.
$KRB5_HOME
This variable is set to point to the installation directory of the Kerberos utilities.
$LOCALWORKDIR
This variable provides access to a local work directory, unique to each individual node used in a batch job, if such a space is provided on the nodes. The local work directory may be implemented in any format as appropriate on each allocated HPC system (e.g. SSD, NVMe, Optane memory, or high speed HDD devices).
$PET_HOME
This variable contains the path to the system-wide accessible directory containing
the tools formerly installed and maintained by the PET staff. Software in $PET_HOME
are considered as experimental or are being evaluated for widespread use. Certain packages in
$PET_HOME will be migrated to $CSE_HOME, as appropriate.
$PROJECTS_HOME
This variable contains the path to the base directory that contains subdirectories for user-supported applications and codes under BC policy FY10-07 (Common Location to Maintain Codes and Shared Data).
$PROJECTS_ARCHIVE
This variable contains the path to the base directory on the local Archive system that contains subdirectories for user-supported applications, code and data under BC policy FY10-07 (Common Location to Maintain Codes and Shared Data).
$SAMPLES_HOME
This variable contains the path to the base directory for the sample codes and scripts installed by a center's staff under BC policy FY06-15 (Sample Code Repository).
$WORKDIR
This is an individual user's directory on the local temporary file system (i.e.,
local high-speed disk) that is available on all HPCMP high performance computing (HPC)
systems. $WORKDIR is intended to be used by executing programs to perform file I/O
that is local to that system in order to avoid slower file I/O across a network mounted
file system, such as a user's home or archive directories. It is not intended to be
used as a permanent file storage area by users. Accordingly, this file system is NOT
backed up or exported to any other system. In the event of file or directory structure
deletion or a catastrophic disk failure, such files and directory structures are lost.
Thus, it is the user's responsibility to transfer files that need to be saved to a
location that allows for permanent file storage, such as the user's archival or home
directory locations.
$WORKDIR2 ($WORKDIR3, etc.)
On systems with additional work file systems beyond the basic $WORKDIR file system,
environment variables corresponding to each added work file system ($WORKDIR2,
$WORKDIR3, etc.) should be added to the default environment of each user with
access to those file systems. If access is conditionally granted, the environment variables
should be instantiated when the user is granted access.
| Date | Revision |
|---|---|
| 03 Apr 2023 | Added $WORKDIR2 (and higher) for systems with multiple work file systems |
| 08 Oct 2021 | BC Team Audit |
| 09 Mar 2021 | Added $PROJECTS_ARCHIVE |
| 19 Oct 2020 | Revised to replace references to retired systems and non-self hosted Phi. Added $CSE_HOME. Added text to $COST_HOME regarding deprecation of COST. |
| 01 Aug 2020 | Added environment variable $LOCALWORKDIR to the policy |
| 27 Sep 2019 | Changed "PETTT" to "PET" |
| 18 Dec 2018 | The policy is modified to explicitly state that all environment variables shall be predefined in the user login, batch and compute environments |
| 20 Sep 2018 | BC Team Audit - Deprecated $CSI_HOME, re-added $PET_HOME as an evaluation area |
| 10 Oct 2017 | Revised definition of $BC_CORES_PER_NODE and introduced the following five environment variables: $BC_NODE_TYPE, $BC_STANDARD_NODE_CORES, $BC_ACCELERATOR_NODE_CORES, $BC_BIGMEM_NODE_CORES and $BC_PHI_NODE_CORES |
| 22 May 2014 | Added $COST_HOME, deprecated $PET_HOME, removed SLM language from $ARCHIVE_HOME and $ARCHIVE_HOST, and stipulated lowercase for hostnames in $BC_HOST |
| 12 Dec 2013 | $DAAC_HOME re-defined |
| 31 May 2012 | $HOST and $KRB5HOME replaced by $BC_HOST and $KRB5_HOME respectively |
| 16 Mar 2012 | BC Team Audit |
| 02 Sep 2011 | $CENTER and $PROJECTS_HOME added to the list |
| 14 Apr 2009 | BC_ Variables added to the list |
| 15 Jan 2009 | $DAAC_HOME added to the list |
| 10 Jul 2008 | $CSI_HOME and $SAMPLES_HOME added to the list |