Cross-Center File Transfers

BC Project: FY06-14
Date of Policy: 19 Oct 2006
Last Updated: 31 Aug 2020 (see Revision Log)

This policy requires three HPCMP-specific cross-center file transfer methods to be available on all allocated HPC systems. Two of the file transfer methods, MPSCP (Multi-Path Secure Copy Program) and SCAMPI (Scalable Copy Accelerated by MPI), employ parallel transfer methods for increased performance, while the third single-threaded method, Tube, enables file transfers without an active user authentication ticket.

Each center shall install the current HPCMP-supported versions of MPSCP, SCAMPI, and Tube on all allocated HPC systems. In the case of Tube, each center shall configure/activate the Tube Server on all allocated HPC systems.

The commands for all three transfer methods must be available in the default path upon login. Detailed descriptions for each of the transfer methods follow.

MPSCP

The Multi-Path Secure Copy Program (MPSCP) is a high-performance file transfer utility that is useful for copying files between systems. MPSCP uses a collection of files and environment variables to provide a standard mode of operation among systems. The files include a required blocked-ports file and an optional configuration file. The centers shall set the environment variables $MPSCP_BLOCKED_FILE and $MPSCP_CONFIG_FILENAME to point to the locations of the respective files.

Click on additional information for more details about the MPSCP command, the blocked-ports and configuration files, and instructions for installation.

SCAMPI

Scalable Copy Accelerated by MPI (SCAMPI) is a command line utility to accelerate data transfer. SCAMPI accelerates data movement through the use of multiple streams over multiple sockets of multiple IPs.

Due to overhead and set up requirements, SCAMPI should only be used when copying more than 1GB among the HPCMP systems. Smaller transfers should use scp or mpscp.

To use SCAMPI click on the link SCAMPI Quick Start Guide.

Tube

Tube is an unattended remote file copy program that uses secure copy to transfer files between hosts. It is intended to support the secure transfer of files from one host to another, even if the user's authentication ticket is not active.

The Tube client specifies a target destination for a file, and the remote host receives the file in the /incoming/user directory, which may be a symbolic link. Files of the same name are not overwritten, therefore the user must remove or rename the file on the remote host before sending another file of the same name.

To use Tube click on the link Tube Quick Start Guide.


Revision Log
Date Revision
31 Aug 2020Added Tube to the policy
10 Jul 2019Added SCAMPI to the policy
19 Apr 2018BC Team Audit
19 May 2016BC Team Audit
10 Mar 2014BC Team Audit
23 Mar 2012BC Team Audit
13 Nov 2008Added file containing blocked ports at participating sites
29 Nov 2007Changed link to Kirby, added other systems