chirp_stream_files1

Usage

Overview

chirp_stream_files is a tool for moving data from one machine to and from many machines, with the option to split or join the file along the way. It is useful for constructing scatter-gather types of applications on top of Chirp.

FCchirp_stream_files copyFT duplicates a single file to multiple hosts. The <localfile> argument names a file on the local filesystem. The command will then open a connection to the following list of hosts, and stream the file to all simultaneously.

FCchirp_stream_files splitFT divides an ASCII file up among multiple hosts. The first line of <localfile> is sent to the first host, the second line to the second, and so on, round-robin.

FCchirp_stream_files joinFT collects multiple remote files into one. The argument <localfile> is opened for writing, and the remote files for reading. The remote files are read line-by-line and assembled round-robin into the local file.

In all cases, files are accessed in a streaming manner, making this particularly efficient for processing large files. A local file name of FC-FT indicates standard input or standard output, so that the command can be used in a pipeline.

Options

-a,--auth=<flag>

Enable authentication mode: unix, hostname, address, ticket, kerberos, or globus.

-b,--block-size=<size>

Set transfer buffer size. (default is 1048576 bytes)

-d,--debug=<flag>

Enable debugging for this subsystem.

-i,--tickes=<files>

Comma-delimited list of tickets to use for authentication.

-t,--timeout=<time>

Timeout for failure. (default is 3600s)

-v,--version

Show program version.

-h,--help

Show help text.

Environment

List any environment variables used or set in this section.

Exit status

On success, returns zero. On failure, returns non-zero.

Examples

To copy the file FCmydataFT to three locations:

% chirp_stream_files copy mydata server1.somewhere.edu /mydata
server2.somewhere.edu /mydata
server2.somewhere.edu /mydata

To split the file FCmydataFT into subsets at three locations:

% chirp_stream_files split mydata server1.somewhere.edu /part1
server2.somewhere.edu /part2
server2.somewhere.edu /part3

To join three remote files back into one called FCnewdataFT:

% chirp_stream_files join newdata server1.somewhere.edu /part1
server2.somewhere.edu /part2
server2.somewhere.edu /part3