Non-Allocated Systems

The HPCMP provides a limited number of HPC systems for non-allocated use. These systems are typically experimental or unique in nature, and do not include a full complement of application software, however they provide users an opportunity to explore nontraditional architectures while operating free of the constraints of allocated use.

Builder is currently Up.

Builder is a single-node Aspen Systems Linux Gigabyte server intended primarily to provide a platform for building and sharing Singularity software containers. The system is populated with AMD 7742 processors and Nvidia V100 graphics processing units. Builder uses Intel Gigabit Ethernet as its high-speed network for I/O traffic. Builder uses AVAGO MegaRAID to manage its local file system that targets 130 TB of disk storage. Builder is equipped with two 64-core processors (128 total cores) running the RHEL 8 operating system, two GPUs, 1,024 gigabytes of memory, with no user-accessible swap space.

More Info

Node Configuration
Login/GPU
Total Nodes 1
Processor AMD 7742 Rome
Processor Speed 2.25 GHz
Sockets / Node 2
Cores / Node 128
Total CPU Cores 128
Usable Memory / Node 1 TB
Accelerators / Node 2
Accelerator NVIDIA V100
Memory / Accelerator 32 GB
Storage on Node 130 TB SAS HDD Array
Interconnect XL710 for 40GbE QSFP+ to cwfs, I350 Gigabit Network Connection to DREN
Operating System RHEL 8.5

Queue Information

Builder does not employ a job scheduler, so there are no queues.

Coral is currently Up.

Coral is an Aspen Systems Linux Cluster with x86_64 and ARM nodes, enabling architecture comparisons. It features 8-way A100 GPUs, AMD MI100/MI250 GPUs, and DPUs for accelerated computing. With standard and large-memory configurations, InfiniBand interconnect, and a Weka parallel file system, Coral supports diverse HPC tasks requiring batch scheduling and parallel processing. It serves as a testbed for ARM-based HPC evaluation.

More Info

Maintenance
Date / TimeDetails
2026 Nov 02 07:30 - Nov 03 00:00 HI (Completed) System Maintenance
Node Configuration
Login Compute Nodes
A100 GPU AMD M100 GPU AMD MI250 GPU ARM + A100 + DPU ARM DPU
Total Nodes 2 8 2 2 4 4 2
Processor Intel 6326 Ice Lake Intel 6338 Ice Lake AMD 7513 Milan AMD 7513 Milan Ampere Altra Q80 Ampere Altra Q64 Intel 6338 Ice Lake
Processor Speed 2.9 GHz 2.0 GHz 2.6 GHz 2.6 GHz 3.0 GHz 3.0 GHz 2.0 GHz
Sockets / Node 2 2 2 2 1 2 2
Cores / Node 32 64 64 64 80 128 64
Total CPU Cores 64 512 128 128 320 512 128
Usable Memory / Node 480 GB 2 TB 1 TB 1 TB 480 GB 480 GB 480 GB
Accelerators / Node 1 8 8 4 2 N/A N/A
Accelerator NVIDIA A40 NVIDIA A100 SXM4 (NVLINK) NVIDIA A100 PCIe AMD MI100 AMD MI250 NVIDIA A100 N/A N/A
Memory / Accelerator 48 GB 80 GB 80 GB 32 GB 128 GB 80 GB N/A N/A
Storage on Node 3.84 TB NVMe SSD 3.84 TB NVMe SSD 3.84 TB NVMe SSD 3.84 TB NVMe SSD 3.84 TB NVMe SSD 6 TB SAS 3.84 TB NVMe SSD 3.84 TB NVMe SSD
Interconnect HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR InfiniBand
Operating System RHEL8 RHEL8 RHEL8 RHEL8 RHEL8 RHEL8 RHEL8 RHEL8
Workspace 240 TBytes WekaIO (all NVMe) + 200 TBytes Object HDD Tier
Node Names coral1
coral2
cr2cn1
cr2cn2
cr2cn3
cr2cn4
cr3cn5
cr3cn6
cr3cn7
cr3cn8
cr4mi1
cr4mi2
cr4mi3
cr4mi4
cr2arm1
cr2arm2
cr2arm3
cr2arm4
cr4arm5
cr4arm6
cr4arm7
cr4arm8
cr1dpu1
cr1dpu2
L3 Cache (MB) N/A 48 48 128 128 32 32 48
Queue Descriptions and Limits on Coral
Priority Queue Name Max Wall Clock Time Max Cores Per Job Max Queued Per User Max Running Per User Description
Highest
Down arrow for decreasing priority debug 30 Minutes N/A N/A N/A Time/resource-limited for user testing and debug purposes
standard 7 Days N/A N/A N/A Standard jobs
transfer 2 Days N/A N/A N/A Data transfer for user jobs. Not charged against project allocation. See the AFRL DSRC Archive Guide, section 5.2.
Lowest
Jean is currently Up.

Jean is a Liqid system located at the ARL DSRC. It has 488 standard compute nodes, 20 large-memory nodes, 42 AI/ML nodes, and 26 visualization nodes.

More Info

Node Configuration
Login Standard Large-Memory Visualization AI/ML Transfer
Total Nodes 9 494 24 24 49 2
Processor Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake
Processor Speed 2.3 GHz 2.3 GHz 2.3 GHz 2.3 GHz 2.3 GHz 2.3 GHz
Sockets / Node 2 2 2 2 2 2
Cores / Node 96^ 96^ 96^ 96^ 96^ 96^
Total CPU Cores 864 47,242 2,304 2,304 2,784 192
Usable Memory / Node 748 GB 385 GB or 757 GB 1546 GB 772 GB 772 GB 772 GB
Accelerators / Node 1 or 4 None None 1 4 None
Accelerator NVIDIA A100 PCIe4 N/A N/A NVIDIA A100 PCIe 4 NVIDIA A100 PCIe 4 N/A
Memory / Accelerator 40 GB N/A N/A 40 GB 40 GB N/A
Storage on Node 7 TB NVMe SSD None 14 TB NVMe SSD 7 TB NVMe SSD 7 TB NVMe SSD N/A
Interconnect HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR InfiniBand HDR Infiniband HDR InfiniBand
Operating System RHEL 8 RHEL 8 RHEL 8 RHEL 8 RHEL 8 RHEL 8

^ There are 96 cores/node. However, WEKA consumes 4 cores/node so only 92 cores/node are available. To ensure your job does not run slowly or encounter issues, 92 available cores/nodes should be used in your batch scripts. For example, if you want to run a 4-node job with all available cores (92 cores/node not 96 cores/node), the following Slurm directives should be utilized: --nodes=4, --ntasks=368 (not 384 cores) and --ntasks-per-node=92 (not 96 cores).

Queue Descriptions and Limits on Jean
Priority Queue Name Max Wall Clock Time Max Cores Per Job Description
Highest urgent 96 Hours 12,000*^ Jobs belonging to DoD HPCMP Urgent Projects
Down arrow for decreasing priority HIE 24 Hours 576*^ Rapid response for interactive work. For more information see the HPC Interactive Environment (HIE) User Guide.
debug** 1 Hour 13,632*^ Time/resource-limited for user testing and debug purposes
high 96 Hours 12,000*^ Jobs belonging to DoD HPCMP High Priority Projects
frontier 168 Hours 12,000*^ Jobs belonging to DoD HPCMP Frontier Projects
cots 96 Hours 12,000*^ Abaqus and Fluent jobs
interactive 12 Hours 12,000*^ Interactive Jobs.
standard 168 Hours 12,000*^ Standard jobs
standard-long 336 Hours 12,000*^ Standard jobs needing additional wall time
AIML 168 Hours 960*^ Dedicated queue for jobs utilizing GPUs
Lowest serial 96 Hours 192*^ User jobs that are not charged against the project allocation

^ Important! Jean is a specialized Mission system for Modeling, Simulation & Analysis and AI/ML work. This system utilizes a flexible access policy to facilitate access for critical, high priority work. All core counts will be adjusted on a "at need" basis determined by the needs of the priority projects running in the priority batch access queues.
* There are 96 cores/node as reflected in the Max Cores per Job column in the table above. However, WEKA consumes 4 cores/node so only 92 cores/node are available. To ensure your job does not run slowly or encounter issues, 92 available cores/nodes should be used in your batch scripts. For example, if you want to run a 4-node job with all available cores (92 cores/node not 96 cores/node), the following Slurm directives should be utilized: --nodes=4, --ntasks=368 (not 384 cores) and --ntasks-per-node=92 (not 96 cores).
** The running job limit on the debug queue per user is 4.

Wheat is currently Up.

Wheat is a Liqid system located at the ERDC DSRC. It has 754 standard compute nodes, 7 large-memory nodes, 16 1-GPU visualization nodes, 83 4-GPU MLA nodes, and 21 6-GPU MLA nodes (a total of 81,052 compute cores). It has 376 TB of useable memory and is rated at 4.2 PFLOPS.

More Info

Node Configuration
Login Standard Large-Memory Visualization 4-GPU MLA 6-GPU MLA
Total Nodes 6 754 7 16 83 21
Processor Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake Intel 9242 Cascade Lake
Processor Speed 2.3 GHz 2.3 GHz 2.3 GHz 2.3 GHz 2.3 GHz 2.3 GHz
Sockets / Node 2 2 2 2 2 2
Cores / Node 92 92 92 92 92 92
Total CPU Cores 552 69,368 644 1,472 7,636 1,932
Usable Memory / Node 8 GB 356 GB 3 TB 731 GB 731 GB 731 GB
Accelerators / Node 1 None 4 1 4 6
Accelerator NVIDIA A100 N/A NVIDIA A100 NVIDIA A100 NVIDIA A100 NVIDIA A100
Memory / Accelerator 40 GB N/A 40 GB 40 GB 40 GB 40 GB
Storage on Node None None None None None None
Interconnect Ethernet InfiniBand InfiniBand InfiniBand InfiniBand InfiniBand
Operating System RHEL 8 RHEL 8 RHEL 8 RHEL 8 RHEL 8 RHEL 8
Queue Descriptions and Limits on Wheat
Priority Queue Name Max Wall Clock Time Max Cores Per Job Description
Highest urgent 168 Hours 2,392 Jobs belonging to DoD HPCMP Urgent Projects
Down arrow for decreasing priority debug^ 1 Hour 3,680 Time/resource-limited for user testing and debug purposes
HIE 24 Hours 92 Rapid response for interactive work. For more information see the HPC Interactive Environment (HIE) User Guide.
high_lw 168 Hours 1,932 Long-walltime jobs belonging to DoD HPCMP High Priority Projects
high_lg 24 Hours 35,420 Large jobs belonging to DoD HPCMP High Priority Projects
frontier_lw 168 Hours 1,932 Long-walltime jobs belonging to DoD HPCMP Frontier Projects
frontier_lg 24 Hours 35,420 Large jobs belonging to DoD HPCMP Frontier Projects
standard_lw 168 Hours 1,932 Long-walltime standard jobs
standard_lg 24 Hours 35,420 Large standard jobs
transfer 48 Hours 1 Data transfer for user jobs. Not charged against project allocation. See the ERDC DSRC Archive Guide, section 5.2.
Lowest background^^ 4 Hours 48 User jobs that are not charged against the project allocation

^ The running job limit on the debug queue per user is 2.
^^ The running job limit on the background queue per user is 3.
Important! Wheat is a Specialized Mission System for Modeling, Simulation & Analysis and AI/ML work. This system utilizes a flexible access policy to facilitate access for critical, high priority work. The service levels are zero-allocated (non-allocated) work running in background queues – available to all users with a zero allocation – while approved priority projects can request priority batch access queues such as standard, high, frontier and urgent.