Cross-Center File Transfers
BC Project: FY06-14
Date of Policy: 19 Oct 2006
Last Updated: 10 Dec 2025 (see Revision Log)
This policy requires the cross-center file transfer utilities, Tube and GridFTP, to be available on all allocated HPC systems. Tube is an HPCMP-specific product that enables single-threaded file transfers without an active user authentication ticket. GridFTP is an extension of the File Transfer Protocol (FTP) specifically designed for grid computing.
Tube
Tube is an unattended remote file copy program that uses secure copy to transfer files between hosts. It is intended to support the secure transfer of files from one host to another, even if the user's authentication ticket is not active.
The Tube client command, tube, must be available in the default path upon login. In addition,
each center shall configure/activate the Tube server process on all allocated HPC systems.
The Tube client specifies a target destination host for a file, and the remote host receives the file in the
/incoming/<username> directory, which may be a symbolic link. Tube does not overwrite files
of the same name; therefore, the user must remove or rename the file on the remote host before sending another
file of the same name.
Click on Tube Quick Start Guide for more details on the use of Tube.
GridFTP (future capability, coming soon)
GridFTP provides a reliable, high-performance file transfer mechanism, particularly for large-scale data transfers in scientific and commercial environments. The HPCMP-supported version takes advantage of the existing Kerberos environment, using ssh to authenticate between systems.
GridFTP has built-in parallelization capabilities that make it particularly well suited for large file transfers (100GB+). Use other transfer utilities such as Tube or scp for smaller files. GridFTP has the capability of resuming failed transfers.
| Date | Revision |
|---|---|
| 10 Dec 2025 | Removed SCAMPI and MPSCP. Added GridFTP. |
| 30 Sep 2021 | BC Team Audit |
| 31 Aug 2020 | Added Tube to the policy |
| 10 Jul 2019 | Added SCAMPI to the policy |
| 19 Apr 2018 | BC Team Audit |
| 19 May 2016 | BC Team Audit |
| 10 Mar 2014 | BC Team Audit |
| 23 Mar 2012 | BC Team Audit |
| 13 Nov 2008 | Added file containing blocked ports at participating sites |
| 29 Nov 2007 | Changed link to Kirby, added other systems |