HPC Centers: Unclassified Systems

When to Contact the HPC Help Desk	Users should contact the HPC Help Desk when assistance is needed for unclassified problems, issues, or questions.
Hours of Operation	8:00 a.m. - 8:00 p.m. Eastern, Monday - Friday (excluding Federal holidays).
HPC Centers Home Page	https://centers.hpc.mil/
Help Desk Video Tutorial	Getting Help: the HPC Help Desk
Phone Number	1-877-222-2039 or (937) 255-0679
Help E-mail	help@helpdesk.hpc.mil
Accounts E-mail	accounts@helpdesk.hpc.mil
HPC Help Desk Manager E-mail	manager@helpdesk.hpc.mil
After Hours	Calls, e-mails and tickets received after normal operating hours will be addressed the following business day.
Mailing Address	DoD HPCMP HPC Help Desk AFRL/RCM 2435 Fifth Street Wright-Patterson Air Force Base, OH - 45433-7802
Ticket Requests	Active users may submit tickets from the Active User Help Ticket. Inactive users may submit tickets using the Inactive User Help Ticket.

Centennial is an SGI ICE XA system located at the ARL DSRC. It has 1,784 standard compute nodes, 32 large-memory compute nodes, and 32 GPU compute nodes (a total of 1,848 compute nodes or 73,920 compute cores). It has 253 TBytes of memory and is rated at 2.6 peak PFLOPS.

More Info

Available Documentation

Node Configuration
	Login Nodes	Compute Nodes
	Login Nodes	Standard Memory	Large Memory	GPU Accelerated
Total Cores \| Nodes	960 \| 24	71,360 \| 1,784	1,280 \| 32	1,280 \| 32
Operating System	RHEL
Cores/Node	40			40 + 1 GPU (1 x 2,880 GPU cores)
Core Type	Intel Xeon E5-2698v4 Broadwell			Intel Xeon E5-2698v4 Broadwell +NVIDIA Tesla K40P
Core Speed	2.2 GHz		2.6 GHz	2.2 GHz
Memory/Node	256 GBytes	128 GBytes	512 GBytes	256 GBytes
Accessible Memory/Node	252 GBytes	124 GBytes	508 GBytes	252 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	Ethernet / InfiniBand

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	debug	1 Hour	N/A	User diagnostic jobs
	transfer	48 Hours	1	Data transfer jobs
	urgent	96 Hours	N/A	Designated urgent jobs by DoD HPCMP
	staff	368 Hours	N/A	ARL DSRC staff testing only. System testing and user support
	high	96 Hours	N/A	Designated high-priority projects by service/agency
	frontier	168 Hours	N/A	Frontier projects only
	cots	96 Hours	N/A	Abaqus, Fluent, and Cobalt jobs
	interactive	12 Hours	N/A	Interactive jobs
	standard-long	200 Hours	N/A	ARL DSRC permission required
	standard	168 Hours	N/A	Normal user jobs
Lowest	background	24 Hours		User jobs that will not be charged against the project allocation

Gaffney is an HPE SGI 8600 system located at the Navy DSRC. It has 704 standard compute nodes, 16 large-memory nodes, and 32 GPU-accelerated nodes (a total of 752 compute nodes or 36,096 compute cores). It has 154 TBytes of memory and is rated at 3.05 peak PFLOPS.

More Info

Available Documentation

Maintenance
Date / Time	Details
2021 Sep 18 14:00 - Sep 19 10:21 CT (Completed)	System Maintenance
2021 Sep 30 08:00 - 18:00 CT	System Maintenance

Node Configuration
	Login Nodes	Compute Nodes
	Login Nodes	Standard Memory	Large Memory	GPU Accelerated
Total Cores \| Nodes	384 \| 8	33,792 \| 704	768 \| 16	1,536 \| 32
Operating System	RHEL
Cores/Node	48			48 + 1 GPU (1 x 3,584 GPU cores)
Core Type	Intel Xeon Platinum 8168			Intel Xeon Platinum 8168 +NVIDIA Tesla P100
Core Speed	2.7 GHz
Memory/Node	384 GBytes	192 GBytes	768 GBytes	384 GBytes +16 GBytes
Accessible Memory/Node	380 GBytes	180 GBytes	744 GBytes	372 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	Intel Omni-Path

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	urgent	24 Hours	768	Designated urgent projects by DoD HPCMP
	frontier	168 Hours	14,400	Designated frontier projects by DoD HPCMP
	high	168 Hours	15,840	Designated high-priority jobs by Service/Agency
	debug	30 Minutes	2,400	User diagnostic jobs
	standard	168 Hours	8,160	Normal priority user jobs
	serial	168 Hours	1	Serial user jobs
	gpu	24 Hours	48	GPU-accelerated jobs
	transfer	48 Hours	N/A	Data transfer jobs
	bigmem	96 Hours	288	Large-memory jobs
Lowest	background	4 Hours	1,200	User jobs that will not be charged against the project allocation

Koehr is an HPE SGI 8600 system located at the Navy DSRC. It has 704 standard compute nodes, 16 large-memory nodes, and 32 GPU-accelerated nodes (a total of 752 compute nodes or 36,096 compute cores). It has 154 TBytes of memory and is rated at 3.05 peak PFLOPS.

More Info

Available Documentation

Maintenance
Date / Time	Details
2021 Sep 18 14:00 - Sep 19 10:21 CT (Completed)	System Maintenance

Node Configuration
	Login Nodes	Compute Nodes
	Login Nodes	Standard Memory	Large Memory	GPU Accelerated
Total Cores \| Nodes	384 \| 8	33,792 \| 704	768 \| 16	1,536 \| 32
Operating System	RHEL
Cores/Node	48			48 + 1 GPU (1 x 3,584 GPU cores)
Core Type	Intel Xeon Platinum 8168			Intel Xeon Platinum 8168 +NVIDIA Tesla P100
Core Speed	2.7 GHz
Memory/Node	384 GBytes	192 GBytes	768 GBytes	384 GBytes +16 GBytes
Accessible Memory/Node	380 GBytes	180 GBytes	744 GBytes	372 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	Intel Omni-Path

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	urgent	24 Hours	768	Designated urgent projects by DoD HPCMP
	frontier	168 Hours	14,400	Designated frontier projects by DoD HPCMP
	high	168 Hours	15,840	Designated high-priority jobs by Service/Agency
	debug	30 Minutes	2,400	User diagnostic jobs
	standard	168 Hours	8,160	Normal priority user jobs
	serial	168 Hours	1	Serial user jobs
	gpu	24 Hours	48	GPU-accelerated jobs
	transfer	48 Hours	N/A	Data transfer jobs
	bigmem	96 Hours	288	Large-memory jobs
Lowest	background	4 Hours	1,200	User jobs that will not be charged against the project allocation

Mustang is currently running in a degraded state.

Mustang is an HPE SGI 8600 system located at the AFRL DSRC. It has 1,128 standard compute nodes, 24 large-memory compute nodes, and 24 GPU compute nodes (a total of 1,176 compute nodes or 56,448 compute cores). It has 244 TBytes of memory and is rated at 4.88 peak PFLOPS.

More Info

Available Documentation

Status: Degraded

Mustang is currently running in a degraded state. Jobs may be negatively affected.

Node Configuration
	Login Nodes	Compute Nodes
	Login Nodes	Standard Memory	Large Memory	GPU Accelerated
Total Cores \| Nodes	576 \| 12	54,144 \| 1,128	1,152 \| 24	1,152 \| 24
Operating System	RHEL
Cores/Node	48			48 + 1 GPU (1 x 3,584 GPU cores)
Core Type	Intel Xeon Platinum 8168			Intel Xeon Platinum 8168 +NVIDIA Tesla P100
Core Speed	2.7 GHz
Memory/Node	384 GBytes	192 GBytes	768 GBytes	384 GBytes +16 GBytes
Accessible Memory/Node	380 GBytes	180 GBytes	744 GBytes	372 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	Intel Omni-Path

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	urgent	168 Hours	28,224	Jobs belonging to DoD HPCMP Urgent Projects
	debug	1 Hour	1,152	User testing
	high	168 Hours	28,224	Jobs belonging to DoD HPCMP High Priority Projects
	frontier	168 Hours	28,224	Jobs belonging to DoD HPCMP Frontier Projects
	standard	168 Hours	28,224	Standard jobs
	HIE	24 Hours	96	Rapid response for interactive work
	transfer	48 Hours	1	Data transfer for user jobs
Lowest	background	120 Hours	48	Unrestricted access - no allocation charge

Narwhal is an HPE Cray EX system located at the Navy DSRC. It has 2,176 standard compute nodes, 12 large-memory nodes, 16 visualization accelerated nodes, 32 1-MLA accelerated nodes, and 32 2-MLA accelerated nodes (a total of 2,268 compute nodes or 290,304 compute cores). It has 590 TBytes of memory and is rated at 12.8 peak PFLOPS.

More Info

Available Documentation

Maintenance
Date / Time	Details
2021 Sep 18 14:00 - Sep 19 10:21 CT (Completed)	System Maintenance
2021 Sep 30 09:00 - 18:00 CT	System Maintenance

Node Configuration
	Login Nodes	Compute Nodes
	Login Nodes	Standard Memory	Large Memory	Visualization Accelerated	1-MLA Accelerated	2-MLA Accelerated
Total Cores \| Nodes	1,408 \| 11	278,528 \| 2,176	1,536 \| 12	2,048 \| 16	4,096 \| 32
Operating System	SLES
Cores/Node	128			128 + 1 GPU (1 x 5,120 CUDA cores, 1 x 640 Tensor cores)		128 + 2 GPUs (2 x 5,120 CUDA cores, 2 x 640 Tensor cores)
Core Type	AMD Epyc 7H12 (64 cores/socket)			AMD Epyc 7H12 +NVIDIA Tesla V100-PCIE
Core Speed	2.6 GHz
Memory/Node	256 GBytes		1 TByte	256 GBytes + 1 x 32 GBytes		256 GBytes + 2 x 32 GBytes
Accessible Memory/Node	246 GBytes		990 GBytes	246 GBytes + 1 x 32 GBytes		246 GBytes + 2 x 32 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	HPE Slingshot

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	urgent	24 Hours	16,384	Designated urgent projects by DoD HPCMP
	frontier	168 Hours	32,768	Designated frontier projects by DoD HPCMP
	high	168 Hours	16,384	Designated high-priority projects by Service/Agency
	debug	30 Minutes	8,192	User diagnostic jobs
	HIE	24 Hours	1,024	Rapid response for interactive work
	viz	24 Hours	128	Visualization jobs
	standard	168 Hours	16,384	Normal priority user jobs
	mla	24 Hours	128	Machine Learning Accelerated jobs
	smla	24 Hours	128	Machine Learning Accelerated jobs
	dmla	24 Hours	128	Machine Learning Accelerated jobs
	bigmem	24 Hours	224	Large-memory jobs
	transfer	24 Hours	N/A	Data transfer jobs
Lowest	background	4 Hours	1,024	User jobs that will not be charged against the project allocation.

Onyx is a Cray XC40/50 system. It has 4,810 standard compute nodes, 4 large-memory compute nodes, 32 GPU compute nodes, 32 Knights Landing (Phi) compute nodes, and 64 Machine Learning Accelerator (MLA) multi-GPGPU nodes (a total of 4,942 compute nodes or 217,128 compute cores). It is rated at 6.06 peak PFLOPS.

More Info

Available Documentation

Node Configuration
	Login Nodes	Compute Nodes
	Login Nodes	Standard Memory	Large Memory	KNL	GPU Accelerated	2-MLA Accelerated	10-MLA Accelerated
Total Cores \| Nodes	264 \| 12	211,640 \| 4,810	176 \| 4	2,048 \| 32	704 \| 32	2,400 \| 60	160 \| 4
Operating System	SLES	Cray Linux Environment				CentOS
Cores/Node	22	44		64	22 + 1 GPU (1 x 3,584 CUDA cores)	40 + 2 GPUs (2 x 5,120 CUDA cores)	40 + 10 GPUs (10 x 5,120 CUDA cores)
Core Type	Dual Intel E5-2699v4 Broadwell (22 cores/socket)			Intel Phi 7230 Knights Landing	Intel E5-2699v4 Broadwell +NVIDIA P100	Dual Intel 6148 Skylake (20 cores/socket) +NVIDIA V100 SXM2	Dual Intel 6148 Skylake (20 cores/socket) +NVIDIA V100 PCIe
Core Speed	2.8 GHz			1.3 GHz	2.8 GHz	2.4 GHz
Memory/Node	256 GBytes	128 GBytes	1 TByte	96 GBytes	256 GBytes +16 GBytes	192 GBytes +2 x 16 GBytes	768 GBytes +10 x 32 GBytes
Accessible Memory/Node	8 GBytes	121 GBytes	991 GBytes	90 GBytes	247 GBytes +16 GBytes	172 GBytes +2 x 16 GBytes	735 GBytes +10 x 32 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	Ethernet	Cray Aries				InfiniBand

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Jobs	Min Cores Per Job	Max Cores Per Job	Comments
Highest	urgent	24 Hours	N/A	22	7,260	Designated urgent jobs by DoD HPCMP
	test	24 Hours	N/A	22	N/A	Staff-only testing
	debug	1 Hour	4	22	11,484	User testing
	HIE	24 Hours	2	22	110	Rapid response for interactive work
	frontier_lg	24 Hours	2	7,261	143,968	Frontier projects only (large jobs)
	frontier_lw	168 Hours	15	22	15,708	Frontier projects only (long walltime)
	frontier_sm	48 Hours	70	22	7,260	Frontier projects only (small jobs)
	high_lg	24 Hours	2	8,449	105,820	Designated high-priority jobs by Service/Agency (large jobs)
	high_lw	168 Hours	15	22	10,824	Designated high-priority jobs by Service/Agency (long walltime)
	high_sm	24 Hours	70	22	8,448	Designated high-priority jobs by Service/Agency (small jobs)
	frontier_md	96 Hours	2	15,709	34,540	Frontier projects only (medium sized, long walltime)
	standard_lg	24 Hours	2	7,261	105,820	Normal priority jobs (large jobs)
	standard_lw	168 Hours	3	22	5,808	Normal priority jobs (long walltime)
	standard_sm	24 Hours	70	22	7,260	Normal priority jobs (small jobs)
	transfer	48 Hours	6	1	1	Data transfer jobs. Access to the long-term storage
Lowest	background	4 Hours	6	22	7,260	Unrestricted access - no allocation charge

SCOUT is an IBM Power9 system located at the ARL DSRC. It has 22 Training nodes, each with 6 nVidia V100 GPUs, 128 Inference nodes, each with 4 nVidia T4 GPUs, and 2 Visualization nodes, each with 2 nVidia GPUs (a total of 152 compute nodes or 6,080 cores). It has 45 TBytes of memory.

More Info

Available Documentation

Node Configuration
	Login Nodes	GPU Accelerated Compute Nodes
	Login Nodes	Training	Inference	Visualization
Total Cores \| Nodes	160 \| 4	880 \| 22	5,120 \| 128	80 \| 2
Operating System	RHEL
Cores/Node	40	40 + 6 GPUs (6 x 5,120 CUDA cores, 6 x 640 Tensor cores)	40 + 4 GPUs (4 x 2,500 CUDA cores, 4 x 320 Tensor cores)	40 + 2 GPUs (2 x 5,120 CUDA cores, 2 x 640 Tensor cores)
Core Type	IBM Power9	IBM Power9 +NVIDIA Volta V100	IBM Power9 +NVIDIA T4 GPUs	IBM Power9 +NVIDIA Volta V100
Core Speed	2.55 GHz
Memory/Node	512 GBytes	700 GBytes	256 GBytes	512 GBytes
Accessible Memory/Node	502 GBytes	690 GBytes	246 GBytes	502 GBytes
Memory Model	Shared on node.	Shared on node. Distributed across cluster.
Interconnect Type	Ethernet / EDR InfiniBand

Queue Descriptions and Limits
Priority	Queue Name	Max Wall Clock Time	Max Cores Per Job	Comments
Highest	transfer	48 Hours	N/A	Data transfer jobs
	urgent	96 Hours	N/A	Designated urgent jobs by DoD HPCMP
	debug	1 Hour	N/A	User diagnostic jobs
	high	168 Hours	N/A	Designated high-priority projects by service/agency
	frontier	168 Hours	N/A	Frontier projects only
	HIE	24 Hours	N/A	Rapid response for interactive work
	interactive	12 Hours	N/A	Interactive jobs
	standard	168 Hours	N/A	Normal user jobs
Lowest	background	24 Hours	N/A	User jobs that will not be charged against the project allocation

HPC Help Desk

Unclassified Systems

Available Documentation

Available Documentation

Available Documentation

Available Documentation

Status: Degraded

Available Documentation

Available Documentation

Available Documentation