allpairs_master1

Usage

Overview

allpairs_master computes the Cartesian product of two sets (<set A> and <set B>), generating a matrix where each cell M[i,j] contains the output of the function F (<compare function>) on objects A[i] (an item in <set A>) and B[j] (an item in <set B>). The resulting matrix is displayed on the standard output, one comparison result per line along with the associated X and Y indices.

allpairs_master uses the Work Queue system to distribute tasks among processors. Each processor utilizes the allpairs_multicore(1) program to execute the tasks in parallel if multiple cores are present. After starting allpairs_master, you must start a number of work_queue_worker(1) processes on remote machines. The workers will then connect back to the master process and begin executing tasks.

Options

-p,--port=<port>

The port that the master will be listening on.

-e,--extra-args=<args>

Extra arguments to pass to the comparison function.

-f,--input-file=<file>

Extra input file needed by the comparison function. (may be given multiple times)

-o,--debug-file=<file>

Write debugging output to this file. By default, debugging is sent to stderr (":stderr"). You may specify logs to be sent to stdout (":stdout") instead.

-O,----output-file=<file>

Write task output to this file (default to standard output)

-t,--estimated-time=<seconds>

Estimated time to run one comparison. (default chosen at runtime)

-x,--width=<item>

Width of one work unit, in items to compare. (default chosen at runtime)

-y,--height=<items>

Height of one work unit, in items to compare. (default chosen at runtime)

-N,--project-name=<project>

Report the master information to a catalog server with the project name - <project>

-P,--priority=<integer>

Priority. Higher the value, higher the priority.

-d,--debug=<flag>

Enable debugging for this subsystem. (Try -d all to start.)

-v,--version

Show program version.

-h,--help

Display this message.

-Z,--port-file=<file>

Select port at random and write it to this file. (default is disabled)

--work-queue-preferred-connection=<connection>

Indicate preferred connection. Chose one of by_ip or by_hostname. (default is by_ip)

Exit status

On success, returns zero. On failure, returns non-zero.

Examples

Let's suppose you have a whole lot of files that you want to compare all to each other, named FCaFT, FCbFT, FCcFT, and so on. Suppose that you also have a program named compareit that when invoked as FCcompareit a bFT will compare files FCaFT and FCbFT and produce some output summarizing the difference between the two, like this:

a b are 45 percent similar

To use the allpairs framework, create a file called FCset.listFT that lists each of your files, one per line:

a
b
c
...

Because allpairs_master utilizes allpairs_multicore(1), so please make sure allpairs_multicore(1) is in your PATH before you proceed.To run a All-Pairs workflow sequentially, start a single work_queue_worker(1) process in the background. Then, invoke allpairs_master.

% work_queue_worker localhost 9123 &
% allpairs_master set.list set.list compareit

The framework will carry out all possible comparisons of the objects, and print the results one by one (note that the first two columns are X and Y indices in the resulting matrix):

1 1 a a are 100 percent similar
1 2 a b are 45 percent similar
1 3 a c are 37 percent similar
...

To speed up the process, run more work_queue_worker(1) processes on other machines, or use condor_submit_workers(1) or uge_submit_workers(1) to start hundreds of workers in your local batch system.

The following is an example of adding more workers to execute a All-Pairs workflow. Suppose your allpairs_master is running on a machine named barney.nd.edu. If you have access to login to other machines, you could simply start worker processes on each one, like this:

% work_queue_worker barney.nd.edu 9123

If you have access to a batch system like Condor, you can submit multiple workers at once:

% condor_submit_workers barney.nd.edu 9123 10
Submitting job(s)..........
Logging submit event(s)..........
10 job(s) submitted to cluster 298.