Non-Allocated Systems

The HPCMP provides a limited number of HPC systems for non-allocated use. These systems are typically experimental or unique in nature, and do not include a full complement of application software, however they provide users an opportunity to explore nontraditional architectures while operating free of the constraints of allocated use.


Talon

The Full System Workbench (Talon) provides HPCMP customers with an opportunity to develop projects that require full control of an HPC system. Each Talon project will work in a dedicated workspace partition and may have sole use of the HPC system when their project is running. Talon projects will be allowed to load and develop codes that aren't allowed in our standard, shared batch environment, such as web interface or database operations. Talon projects will coordinate with the AFRL DSRC support team to have their special software applications installed.

More

Available Documentation

talon.afrl.hpc.mil
Aspen Systems Intel - 13.5 TFLOPS
Login Nodes Compute Nodes
Project
(dedicated)
Standard Memory Accelerator Nodes
GPU Phi
Total Nodes 1 12 16 1 1
Operating System RHEL 6.7
Cores/Node 16 24 24 + 2 GPU
(2 x 4,992 GPU cores)
24 + 2 Phi
(2 x 61 Phi cores)
Core Type Intel E5-2640v3
+NVIDIA Grid K1
Intel E5-2640v3 Intel E5-2670v3 Intel E5-2670v3
+NVIDIA Tesla K80
Intel E5-2670v3
+Intel 7120P
Core Speed 2.6 GHz 2.3 GHz
Memory/Node 128 GBytes
+16 GBytes
128 GBytes 128 GBytes
+48 GBytes
128 GBytes
+32 GBytes
Accessible Memory/Node 125 GBytes
Memory Model Shared on node. Shared on node.
Distributed across cluster.
Interconnect Type 4x FDR InfiniBand

Talon consists of the following:

  • 32 user-accessible nodes each with dual 2.6-GHz, eight-core Haswell processors (16 Cores/Node)
    • 1 login or web nodes with 128 GBytes RAM and 10-Gig Ethernet connections to DREN
    • 1 dedicated transfer node with 128 GBytes RAM and 10-Gig Ethernet connections to DREN
    • 30 compute nodes with 128 GBytes RAM
    • 131.84-TByte high-bandwidth (220 GBytes/sec) Panasas parallel file system
  • 33-TByte high-bandwidth (1.2 GBytes/sec) Panasas parallel file system
Summary of Queues on the Aspen Systems Intel - Talon
Priority Queue
Name
Job
Class
Max Wall
Clock Time
Max Cores
Per Job
Comments
Highest debug Debug 1 Hour N/A User diagnostic jobs.
Down Arrow for decreasing priority urgent Urgent N/A N/A Designated High Urgent jobs by DoD HPCMP
frontier Frontier 168 Hours N/A Jobs belonging to DoD HPCMP Frontier Projects
high High N/A N/A Jobs belonging to DoD HPCMP High Priority Projects
standard Standard 96 Hours N/A Non Challenge User Jobs
background Background 24 Hours N/A No allocation charge
Lowest transfer N/A 24 Hours 1 Transfer Jobs

Users

Talon is installed on the AFRL DSRC production network and is open to all users in the HPCMP domain. Any non-HPCMP user interested in accessing Talon will need to complete the HPCMP user account process with his organizational S/AAA.

Obtaining an Account

Authorized DOD and Contractor personnel may request an account by submitting a proposal to the AFRL DSRC via email to sp-proposal@helpdesk.hpc.mil. The proposal should include the following information:

  • HPC experience and level of required support
  • Project suitability for a Shared Memory system
  • Project contribution to the DoD mission and/or HPC technical advancement
  • Proposed workload

Direct any questions regarding these non-allocated systems to sp-proposal@helpdesk.hpc.mil.

Project Proposals

Any user may submit a project proposal at any time and our review team is committed to providing a response within two weeks. A copy of the call for proposals is available here. Users with routine requests to run known shared memory codes may limit their proposals to identifying the codes needed and a very brief description of memory requirements and job types.

Project Approval

Project proposals can be submitted to the HPC Help Desk at any time, to a special projects mailbox:
sp-proposal@helpdesk.hpc.mil

Project Support

With a limited number of users sharing a small number of nodes on a system, good collaboration is critical. All system users will be added to a system mailing list, monitored by our support staff. Users will be required to post notices of their plans for system usage over certain thresholds. This will alert the support staff and other users as system load is impacted.

Current plans for Talon call for a configuration as similar to allocated HPC systems as possible, including the use of PBS. We will run one queue with no limits.

Call for Non-Allocated Project Proposals

AFRL DSRC through the High Performance Computing (HPC) Modernization Program (HPCMP) is now offering access to special purpose, non-allocated systems. HPCMP understands that not all HPC requirements can be satisfied with our current offering of large HPC clusters running in a shared batch environment. One system is now available to all current or future HPCMP users. A Full System Workbench, named TALON, with 32 nodes, each with 16 cores and 128 GBytes of memory.

To apply for participation, Principal Investigators (PI) should prepare a short proposal (no more than two pages) containing the following:

  • PI's name and contact information
  • Organizational S/AAA and contact information
  • Name of your application code
  • Identify system to be used
  • A brief discussion of the following:
    • Explain why application or project won't run effectively on a standard HPCMP allocated system.
    • Estimate usage to include:
      • an overall estimate of time needed, both in core-hours and overall project duration
      • the length of time in wallclock hours for a typical production run
      • the number and size of production jobs
    • Names of additional users who will be participating in this project
    • Project's impact to include a brief description how this project contributes to DoD mission and/or HPC technical advancement.

Projects will be required to provide brief progress reports and/or briefings on a regular basis. Our goal is to document your success stories to demonstrate good return on investment during challenging budget reviews.

This is an ongoing effort; proposals can be submitted at any time. The Special Projects review team will provide a response to submitted proposals within two weeks. Schedule and amount of time available is subject to change depending on overall demand and priorities. Participants are required to keep the support team informed of their near-term schedules and system plans.

Proposals should be submitted to your Organizational S/AAA and directly to the Special Projects review team at sp-proposal@helpdesk.hpc.mil. Any questions should also be submitted to this e-mail address.

Hokulea

Hokule'a is an IBM POWER8 server with high-performance, high-bandwidth, NVIDIA GP100 graphical processing units (GPUs). Utilizing the GPUs, Hokule'a will provide approximately 690 TFLOPs of supercomputing. As its name implies, it will be used as a test system to evaluate the performance of this novel architecture for DoD-specific software.

More

hokulea.mhpcc.hpc.mil
IBM POWER8 - 690 TFLOPS
Login Nodes GPU Accelerated Compute Nodes
Total Cores | Nodes | GPUs 20 | 1 | 0 640 | 32 | 128
Operating System RHEL 7
Cores/Node 20 20 + 4 GPU
(4 x 3,538 GPU cores)
Core Type IBM POWER8 IBM POWER8
+NVIDIA Tesla P100
Core Speed 2.86 GHz
Memory/Node 256 GBytes 256 GBytes
+(4 x 16 GBytes)
Accessible Memory/Node 240 GBytes 244 GBytes
+(4 x 16 GBytes)
Memory Model Distributed
Interconnect Type Mellanox SB7700 36 X EDR 100 Gbits/sec InfiniBand
Workspace 250 TBytes
Summary of Queues on the IBM POWER8 - Hokule'a
Priority Queue
Name
Job
Class
Max Wall
Clock Time
Max Cores
Per Job
Comments
Highest debug Debug 1 Hour N/A User diagnostic jobs.
Down arrow for decreasing priority urgent Urgent N/A N/A Designated High Urgent jobs by DoD HPCMP
high High N/A N/A Jobs belonging to DoD HPCMP High Priority Projects.
frontier Frontier 168 Hours N/A Jobs belonging to DoD HPCMP Frontier Projects.
standard Standard 96 Hours N/A Normal Priority User Jobs
Lowest background Background 24 Hours N/A User jobs that will not be charged.

System Objective and Intended Use

MHPCC's goal is to inform future HPCMP Technology Insertion by assessing the performance of the Power8+NVIDIA system with high pay-off applications for the DoD. The MHPCC DSRC is seeking collaborators to help us meet that objective. Specifically, benchmarking comparisons of applications on production HPC systems and the Power8+NVIDIA system are sought. Account access and PBS queue priority will be given to projects that help us meet this objective.

With a limited number of users sharing a small number of nodes, consideration is critical. All system users will be added to a system mailing list, monitored by our support staff. Please understand that system administrators may restrict access to the system to allow for key benchmarking activities. Any such restrictions will be preceded by notice to all account holders via the system mailing list.

Obtaining an Account

Any user with an active account in the Portal to the Information Environment (pIE) may request an account at any time, and our review team is committed to providing a response within two weeks. To apply for participation, Principal Investigators (PI) should prepare a short access request. Please follow the attached template: MHPCC Special HPC Project

Project Support

Requests and questions can be submitted to the HPC Help Desk's Vanguard Center special projects mailbox: proposal@mhpcc.hpc.mil

Project Proposals

Proposals can be submitted at any time. The Vanguard Center review team will provide a response to submitted access request within two weeks.