Cross-Center File Transfers
BC Project: FY06-14
Date of Policy: 19 Oct 2006
Last Updated: 30 Sep 2021 (see Revision Log)
This policy requires three HPCMP-specific cross-center file transfer utilities to be available on all allocated HPC systems. Two of the utilities, MPSCP (Multi-Path Secure Copy Program) and SCAMPI (Scalable Copy Accelerated by MPI), employ parallel transfer methods for increased performance, while the third utility, Tube, enables single-threaded file transfers without an active user authentication ticket.
Each center shall install the current HPCMP-supported versions of MPSCP, SCAMPI, and Tube on all allocated HPC systems. In the case of Tube, each center shall also configure/activate the Tube Server on all allocated HPC systems.
The commands for all three transfer utilities must be available in the default path upon login. Detailed descriptions for each of the transfer utilities follow.
MPSCP
The Multi-Path Secure Copy Program (MPSCP) is a high-performance file transfer utility
that is useful for copying files between systems. MPSCP uses a collection of files and
environment variables to provide a standard mode of operation among systems. The files
include a required blocked-ports file and an optional configuration file. The centers
shall set the environment variables $MPSCP_BLOCKED_FILE and
$MPSCP_CONFIG_FILENAME to point to the locations of the respective files.
Click on additional information for more details about the MPSCP command, the blocked-ports and configuration files, and instructions for installation.
SCAMPI
Scalable Copy Accelerated by MPI (SCAMPI) is a command line utility to accelerate data transfer. SCAMPI accelerates data movement through the use of multiple streams over multiple sockets of multiple IPs.
Due to overhead and set up requirements, SCAMPI should only be used when copying more
than 1GB among the HPCMP systems. Smaller transfers should use scp or
mpscp.
Click on SCAMPI Quick Start Guide for more details on the use of SCAMPI.
Tube
Tube is an unattended remote file copy program that uses secure copy to transfer files between hosts. It is intended to support the secure transfer of files from one host to another, even if the user's authentication ticket is not active.
The Tube client specifies a target destination for a file, and the remote host receives
the file in the /incoming/<username> directory, which may be a symbolic link. Files
of the same name are not overwritten, therefore the user must remove or rename the file on
the remote host before sending another file of the same name.
Click on Tube Quick Start Guide for more details on the use of Tube.
| Date | Revision |
|---|---|
| 30 Sep 2021 | BC Team Audit |
| 31 Aug 2020 | Added Tube to the policy |
| 10 Jul 2019 | Added SCAMPI to the policy |
| 19 Apr 2018 | BC Team Audit |
| 19 May 2016 | BC Team Audit |
| 10 Mar 2014 | BC Team Audit |
| 23 Mar 2012 | BC Team Audit |
| 13 Nov 2008 | Added file containing blocked ports at participating sites |
| 29 Nov 2007 | Changed link to Kirby, added other systems |