Cross-Center File Transfers
BC Project: FY06-14
Date of Policy: 19 Oct 2006
Last Updated: 31 Aug 2020 (see Revision Log)
This policy requires three HPCMP-specific cross-center file transfer methods to be available on all allocated HPC systems. Two of the file transfer methods, MPSCP (Multi-Path Secure Copy Program) and SCAMPI (Scalable Copy Accelerated by MPI), employ parallel transfer methods for increased performance, while the third single-threaded method, Tube, enables file transfers without an active user authentication ticket.
Each center shall install the current HPCMP-supported versions of MPSCP, SCAMPI, and Tube on all allocated HPC systems. In the case of Tube, each center shall configure/activate the Tube Server on all allocated HPC systems.
The commands for all three transfer methods must be available in the default path upon login. Detailed descriptions for each of the transfer methods follow.
The Multi-Path Secure Copy Program (MPSCP) is a high-performance file transfer utility
that is useful for copying files between systems. MPSCP uses a collection of files and
environment variables to provide a standard mode of operation among systems. The files
include a required blocked-ports file and an optional configuration file. The centers
shall set the environment variables
$MPSCP_CONFIG_FILENAME to point to the locations of the respective files.
Click on additional information for more details about the MPSCP command, the blocked-ports and configuration files, and instructions for installation.
Scalable Copy Accelerated by MPI (SCAMPI) is a command line utility to accelerate data transfer. SCAMPI accelerates data movement through the use of multiple streams over multiple sockets of multiple IPs.
Due to overhead and set up requirements, SCAMPI should only be used when copying more
than 1GB among the HPCMP systems. Smaller transfers should use
To use SCAMPI click on the link SCAMPI Quick Start Guide.
Tube is an unattended remote file copy program that uses secure copy to transfer files between hosts. It is intended to support the secure transfer of files from one host to another, even if the user's authentication ticket is not active.
The Tube client specifies a target destination for a file, and the remote host receives
the file in the
/incoming/user directory, which may be a symbolic link. Files
of the same name are not overwritten, therefore the user must remove or rename the file on
the remote host before sending another file of the same name.
To use Tube click on the link Tube Quick Start Guide.
|31 Aug 2020||Added Tube to the policy|
|10 Jul 2019||Added SCAMPI to the policy|
|19 Apr 2018||BC Team Audit|
|19 May 2016||BC Team Audit|
|10 Mar 2014||BC Team Audit|
|23 Mar 2012||BC Team Audit|
|13 Nov 2008||Added file containing blocked ports at participating sites|
|29 Nov 2007||Changed link to Kirby, added other systems|