Welcome to the HPC Help Desk video series.
This presentation will introduce you to the Scalable Copy
Accelerated by Message Passing Interface, commonly known
as SCAMPI.
This presentation is based on a tutorial
written by Jacob Cohen of the U.S. Army Combat
Capabilities Development Command of the Army Research
Laboratory, the SCAMPI Overview at the ARL DSRC website and the
SCAMPI Quick Start Guide found on the HPCMP centers.hpc.mil
website. When you see this graphic and hear this sound,
it means the information is contained in the accompanying job aid,
which includes all the URLs, phone numbers and email
contacts referenced in this video.
Please note, this tutorial covers SCAMPI and its variants,
but does not directly address Tube or MPSCP and other common
file transfer utilities available on the HPC systems
Due to the setup and overhead costs of launching MPI
SCAMPI is not recommended for transfer smaller than one gigabyte
total. The available methods for file transfers are found
in each system's user guide under section 3.3.
We will begin with an overview of the SCAMPI utility,
how it works,
how it compares to other transfer methods
and where you can find the file.
Next, we'll look at specific parameters which define
how many processes will be used on which machines.
We'll take a look at two variants of SCAMPI
and their specialized purposes.
And finally, we will review tips for more efficient transfers.
SCAMPI and its variants take advantage of multiple MPI processes
by splitting single large files into multiple transfer streams,
as well as running
multiple small file transfers concurrently.
For large directory or file transfers,
SCAMPI's parallelization can significantly decrease transfer time
when compared to regular SCP transfer.
On an HPC system, SCAMPI
will be located in the /app/mpiutil directory.
This directory contains the run scripts for SCAMPI and its variants,
as well as library dependencies and other files required for
SCAMPI's MPI code to run.
Running SCAMPI's help feature,
we can see a split of arguments between MPI flags,
optimization and debugging, encryption and other.
Let's start with the MPI flags, as these have the most impact
on how SCAMPI launches MPI and performs its transfer.
-nh sets the number of host systems or total
login nodes on which SCAMPI MPI processes will be launched.
This number must be at least two and also must be divisible
by two for an equal number of sending and receiving hosts.
In the minimum case of -nh2, one host contains the files
to transfer, and
one host will be receiving these files. In a case of
nh 6, three hosts will be sending and three will be receiving.
-pph, or processes per host, determines
how many MPI processes or threads will run on each of the hosts
used in the transfer.
-pph must be a number greater than one as
SCAMPI will not run on serial transfers. Together with -nh
We get -nh2, -pph2.
These are the smallest parameters for SCAMPI transfers,
with each host running two processes to send and receive.
Since SCAMPI uses paired processes to send and receive data.
The number of data streams is host times threads divided by two.
So for this two host,
two process per host transfer
SCAMPI will be running with two data streams.
Let's look
at an optimization flag for transferring a number of smaller files.
The -m flag allows you to set the threshold
in megabytes for what SCAMPI considers small or large.
By default, this value is set to one gigabyte.
So SCAMPI would send a 750 megabyte file regularly
without parallel file splitting as it is smaller than the threshold.
However, if you set -m to 500, the 750
megabyte file will run as a faster multi-threaded transfer.
Let's look at a simple transfer from the user system
to a remote HPC system.
This example transfers a file we'll call "myfile"
from the home directory
on the local system to the home directory on the HPC system,
using two nodes with five threads each, one on the sending system
and one on the receiving system.
In this example, there will be five streams of data.
Now, a more complicated example,
here we are running SCAMPI across four hosts with six processes
per host.
We also specify that multi-threaded transfer should occur
if a file to be transferred is larger than 400 megabytes.
The -vvv command specifies the highest level
of verbosity or description detail to be given.
We will be completing a directory transfer from the remote system
to our local received directory.
The SCAMPI command-line order is always sender, then receiver,
even if the sender is the remote system.
Now that we are familiar with base SCAMPI execution,
let's look into the variants that are available on HPCMP systems.
First is internal SCAMPI or iSCAMPI.
iSCAMPI should be used for internal runs, a batch jobs on a system
as there is no external communication or authentication.
iSCAMPI is commonly used in line and batch jobs
and typically run on a subset of job allocated nodes.
Processes in iSCAMPI both read and write data,
which means the number of data streams is host times threads.
Call the run script "iscampi."
Like before use, -nh
to specify the total number of hosts you wish to use and -pph
to specify the number of processes you wish for each host
to run, with a maximum of 128 threads.
The remaining syntax is modeled after the SCP command.
However, there is no host information allowed as nodes will be
selected from the batch pool in your job.
The -l option should always be used when targeting
Lustre file systems, so striping parameters will be invoked.
The -s option will provide a summary histogram of the files.
This example transfers from the $home directory to $workdir
on the local system,
using four nodes with six threads for a total of 24 data streams.
Hybrid SCAMPI, or HiSCAMPI,
is the other variant made available on HPCMP systems.
HiSCAMPI is meant to be run
across login nodes when moving files to and from $CENTER
or $ARCHIVE into or out of an HPC system directory,
such as $HOME, $WORKDIR or $PROJECT.
HiSCAMPI is very similar to iSCAMPI, but with parallel NFS file
system handling. Both variants follows similar command
line formats when compared to regular SCAMPI.
This transfer from $CENTER on the local system to $WORKDIR
on the local system uses four nodes with six threads.
In this case, $WORKDIR
is a Lustre file system, so -l is used.
-s provides a summary histogram of the files using a shared
version of iSCAMPI.
Finally, here are
some tips to improve your transfers with SCAMPI.
First, note that SCAMPI may require manual approval of ssh
connections during setup.
Due to the setup and overhead costs of launching MPI, SCAMPI
is not recommended for transfers smaller than one gigabyte total.
Having more than eight processes per node
has been shown to give diminishing returns on transfer times.
Finally, it is not uncommon for SCAMPI
to fail during initial setup due to busy or down remote login nodes.
Multiple startup failures should be reported to the HPC
Help Desk.
Thank you for taking the time to view this instructional video. If
you have any additional questions, please contact the HPC Help
Desk. The phone numbers, email address and HPC Help
Desk Web Portal information is shown here
and can also be found in the accompanying job aid.