Common Node Attribute Specification

BC Project: FY14-01
Date of Policy: 13 Mar 2014
Last Updated: 08 Dec 2020 (see Revision Log)

This project defines a common method for specifying the type of nodes required on all allocated HPC systems with heterogeneous node configurations.

Definition of a common method will facilitate commonality across HPC Centers and systems.

Currently, the centers use Workload Management System directives to specify node attributes. The Altair PBS Professional #PBS –l select directive is used here. Examples of node attributes are memory size, or presence of an accelerator such as an Intel Many Integrated Core (MIC), also known as XEON Phi or a Graphics Processing Unit (GPU). The following generic names will be used to specify the listed attributes:

Node Attribute Specification
Attribute Description
bigmem A node which has more memory than other nodes.
nmics Number of Intel MIC (phi) devices per node
ngpus Number of Nvidia GPU devices per node

The list of attributes may increase as the use of accelerators in high performance computing increases.

Request for 4 nodes with 48 CPUs (no special devices):

#PBS -l select=4:ncpus=48:mpiprocs=1

Request for 2 bigmem nodes with 44 CPUs:

#PBS -l select=2:ncpus=44:bigmem=1:mpiprocs=1

Request for 8 nodes with 64 CPUs and 1 nmic:

#PBS -l select=8:ncpus=64:nmics=1:mpiprocs=1

Request for 16 nodes with 48 CPUs and 1 GPU:

#PBS -l select=16:ncpus=48:ngpus=1:mpiprocs=1

Requests for 1 node with 44 CPUs (no special devices) and 32 nodes with 22 CPUs and 1 GPU:

#PBS -l select=1:ncpus=44:mpiprocs=1+32:ncpus=22:ngpus=1:mpiprocs=1

A summary of usage for each system for the participating HPCMP centers can be found at the HPC Centers website in the "Guide to the Batch Queuing System."

Notice: Future HPCMP allocated HPC systems may make use of a diversity of HPC job schedulers (PBSpro, SLURM, LSF) to submit batch jobs. Accordingly, hereafter revisions of this policy will include examples of how other job schedulers make use of directives to specify node attributes supported by the job schedulers.

Revision Log
Date Revision
08 Dec 2020Replaced PBS scheduler by a generic HPC job scheduler
26 Apr 2018BC Team Audit
16 Jun 2016BC Team Audit
13 Mar 2014First release of policy