Data Pool Rsync Administration
This page is under construction.
The rsync daemon has access to one or more sets of data pools. Each set of pools is associated with an rsync module name. For example, you could have a set of pools associated with the module name PV and another set with the module name IST. You can also use module names with subsets of larger pools. For example, if you have a root directory for the pools created during each week of PV, you could have module names PV1, PV2, and so on. The generic PV module could then be associated with the directory above all the weekly pool directories. The installation would look something like this:
pools | |-IST | |-PV | |-PV1 | |-PV2 | |-PV...
You'll find an example showing how module names and directories are related in the configuration file.
When a client makes a request of the daemon, it uses a module name, so it could request all PV pools, or just those for a specific week, for example, requesting module PV2.
As pools are created and updated, clients continually make requests of the daemon. These requests make a mirror of the ICC module at the client site.
The daemon allows an authorized client to get lists of items and to copy directories, files and symbolic links. It does not allow the client to write into the module area.
Clients are authorized in two ways:
The client must supply a password to the daemon. This password is associated with a user name and password in the daemon's rsyncd.secrets file. Authorized users can be allowed access to all modules or just to some.
The machines and subnets that can connect to the daemon are listed in the rsyncd.config file by IP address or domain name. As with user names, you can limit module access to only selected machines or subnets.
Currently, traffic between the daemon and a client is not encrypted; but that can be added.
The rsync documentation describes how to set-up an rsync daemon. In this section, we give a basic example. Our example does not run as root, so we explicitly start it and give it it's own TCP port number.
Use a machine that has rynsc installed, or install it yourself. Check for installation with the command
> rsync --version rsync version 2.6.3 protocol version 28 Copyright (C) 1996-2004 by Andrew Tridgell and others <http://rsync.samba.org/> Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, inplace, IPv6, 64-bit system inums, 64-bit internal inums
Once you have the software installed, you'll need to create some configuration files:
rsyncd The script used to start the daemon manually.
rsyncd.config The configuration file read by the rsync daemon.
rsyncd.secrets The file containing the list of authorized users.
When you run the daemon, it will create some additional files:
rsyncd.lock This supports the maximum connections option specified in the rsyncd.config file.
rsyncd.log Errors and authorization failures are written to this file.
rsyncd.pid This file contains the daemons process ID.
Here's a simple shell script to start the rsync daemon as a non-root process. The command takes three parameters:
- Run rsync as a daemon, or server, process.
- Listen on TCP port 44520.
Read the daemon's configuration from file /home/poolman/rsync/rsyncd.config
Start the process in the background and then wait 5 seconds. After that, check that the server is indeed running.
The script's file protection is set such that the owner can execute the file.
File name: rsyncd
# File rsyncd echo "Starting rsyncd `date`" rsync --daemon --port=44520 --config=/home/poolman/rsync/rsyncd.config & sleep 5 ps -elf | grep 'rsync --daemon' | grep -v grep
The Configuration File
The rsync daemon's cofiguration commands are described in rsyncd.conf. Here's an example configuration file. In this configuration, most options are global. Module names are give in square brackets. Indented items below the module name, including the root directory for the module, are given below the name.
File name: rsyncd.config
# File: rsyncd.config # rsync daemon command # rsync --daemon --port=44520 --config=/home/poolman/rsync/configd.rsync secrets file = /home/poolman/rsync/rsyncd.secrets # If you want something to appear when someone logs on, put it in the # rsyncd.motd file #motd file = /home/poolman/rsync/rsyncd.motd lock file = /home/dppool/rsync/rsyncd.lock pid file = /home/dppool/rsync/rsyncd.pid # Use if run by root #use chroot = yes read only = yes list = yes uid = poolman gid = users max connections = 16 # Could use syslogs if running as root log file = /home/dppool/rsync/rsyncd.log #log format = %t %a %m %f %b timeout = 300 auth users = mpe, nhsc, leuven saclay marseille liege wien hosts allow = herschel6.kluvent.be hcss1.ipac.caltech.edu 139.8.289.0/8 165.3.183/8 194.43.294.0/8 188.8.131.52/8 184.108.40.206/8 hosts deny = * [PV] comment = All PV obsid pools path = /pools/pv [PV1] comment = Week 1 pools path = /pools/pv/pv1 [PV2] comment = Week 2 pools path = /pools/pv/pv2 [PVCAL] comment = Calibration files contributed by PACS calibration scientists. path = /cal/pv/special
The user names and passwords of agents allowed to access the server are kept in the rsyncd.secrets file. The file protections are set so only the owner can read and write the file.
File name: rsyncd.secrets
mpe:password1 leuven:password2 nhsc:password3 saclay:password4 marseille:password5 liege:password6 wien:password7