Copying Pools with Rsync
General
*[#install Software installation]
*If you have questions, contact MailTo(John DOT Rector AT ipac DOT caltech DOT edu)
*Server administrators should refer to [wiki:PACS/RsyncAdmin Data Pool Rsync Administration]
Overview
Level 0 product pools are created from data queried from the ICC's Versant database and stored as product pools on a computer accessible by other internal and extern nodes, including machines at Sub-ICC institutes. The ICC, where the pools are created and maintained, runs an rsync daemon having assess to all product pools. A client rsync installation is used to copy the pools.
Sets of related product pools are stored in rsync modules that are given meaningful names to data pool. For example, the module name "PV" would contain all PV pools. Using this module name you could list or get all, or particular sets, of files. Module names are also used for subsets of data pools. For example, you might have names like "PV<obsid>,..." or "PV<sequence number>,..." or "PV<start date>,...".
transferPools
transferPools is a script used for making the copies. It can perform the following functions.
# List the rsync modules. # List all or a hierarchical subset of files within a module. # Copy and update all or a subset of files in a module. # Run as a CRON job at regular intervals, copying and updating files as data pools are created and updated at the ICC.
Command Syntax
transferPools test:: Use this command to test your configuration. The command creates the sub-directories and files used for running transferPools as a CRON job and then run the rsync command but don't actually copy the files. (You will learn more about what's created below.)
transferPools list:: List the names of the data pool modules and their descriptions.
transferPools list <module>[<path>]:: List the files with a pool. If a path is included as well, the listing is limited to those items in that path.
transferPools copy <module>[<path>] <local path>:: Use this command to make a single copy of all the files in the pool, or those under the module path if a path is specified. (By default transferPools creates log files and adds other infrastructure support. The copy option prevents the infrastructure from being created and used. Pools are copied from the module to the local path destination to specify.)
transferPools <module>[<path>] <local path>:: Use this command to copy all data pools in a modules as a CRON job that continues to run at frequent intervals.
Running transferPools as a CRON Job
Examples
1. Display the command syntax.
> ./transferPools
Usage: transferPools test
transferPools <module>
transferPools list [<module>]
transferPools get <module>[<path>] <local path>
tansferPools <module>[<path>] <local path>