Data Pool Rsync Administration


This page is under construction.


The rsync daemon has access to one or more sets of data pools. Each set of pools is associated with an rsync module name. For example, you could have a set of pools associated with the module name PV and another set with the module name IST. You can also use module names with subsets of larger pools. For example, if you have a root directory for the pools created during each week of PV, you could have module names PV1, PV2, and so on. The generic PV module could then be associated with the directory above all the weekly pool directories. The installation would look something like this:


You'll find an example showing how module names and directories are related in the configuration file.

When a client makes a request of the daemon, it uses a module name, so it could request all PV pools, or just those for a specific week, for example, requesting module PV2.

As pools are created and updated, clients continually make requests of the daemon. These requests make a mirror of the ICC module at the client site.


The daemon allows an authorized client to get lists of items and to copy directories, files and symbolic links. It does not allow the client to write into the module area.

Clients are authorized in two ways:

  1. The client must supply a password to the daemon. This password is associated with a user name and password in the daemon's rsyncd.secrets file. Authorized users can be allowed access to all modules or just to some.

  2. The machines and subnets that can connect to the daemon are listed in the rsyncd.config file by IP address or domain name. As with user names, you can limit module access to only selected machines or subnets.

Currently, traffic between the daemon and a client is not encrypted; but that can be added.

Software Installation

The rsync documentation describes how to set-up an rsync daemon. In this section, we give a basic example. Our example does not run as root, so we explicitly start it and give it it's own TCP port number.

Use a machine that has rynsc installed, or install it yourself. Check for installation with the command

> rsync --version
rsync  version 2.6.3  protocol version 28
Copyright (C) 1996-2004 by Andrew Tridgell and others
Capabilities: 64-bit files, socketpairs, hard links, symlinks, batchfiles, 
              inplace, IPv6, 64-bit system inums, 64-bit internal inums

Once you have the software installed, you'll need to create some configuration files:

When you run the daemon, it will create some additional files:

Daemon Startup

Here's a simple shell script to start the rsync daemon as a non-root process. The command takes three parameters:

  1. Run rsync as a daemon, or server, process.
  2. Listen on TCP port 44520.
  3. Read the daemon's configuration from file /home/poolman/rsync/rsyncd.config

Start the process in the background and then wait 5 seconds. After that, check that the server is indeed running.

The script's file protection is set such that the owner can execute the file.

File name: rsyncd

# File rsyncd

echo "Starting rsyncd `date`"
rsync --daemon --port=44520 --config=/home/poolman/rsync/rsyncd.config &
sleep 5
ps -elf | grep 'rsync --daemon' | grep -v grep

The Configuration File

The rsync daemon's cofiguration commands are described in rsyncd.conf. Here's an example configuration file. In this configuration, most options are global. Module names are give in square brackets. Indented items below the module name, including the root directory for the module, are given below the name.

File name: rsyncd.config

# File: rsyncd.config

# rsync daemon command
# rsync --daemon --port=44520 --config=/home/poolman/rsync/configd.rsync

secrets file = /home/poolman/rsync/rsyncd.secrets
# If you want something to appear when someone logs on, put it in the
# rsyncd.motd file
#motd file = /home/poolman/rsync/rsyncd.motd
lock file = /home/dppool/rsync/rsyncd.lock
pid file = /home/dppool/rsync/
# Use if run by root
#use chroot = yes
read only = yes
list = yes
uid = poolman
gid = users
max connections = 16
# Could use syslogs if running as root
log file = /home/dppool/rsync/rsyncd.log
#log format = %t %a %m %f %b
timeout = 300
auth users = mpe, nhsc, leuven saclay marseille liege wien
hosts allow = 165.3.183/8
hosts deny = * 

   comment = All PV obsid pools
   path = /pools/pv

   comment = Week 1 pools
   path = /pools/pv/pv1
   comment = Week 2 pools
   path = /pools/pv/pv2

   comment = Calibration files contributed by PACS calibration scientists.
   path = /cal/pv/special


The user names and passwords of agents allowed to access the server are kept in the rsyncd.secrets file. The file protections are set so only the owner can read and write the file.

File name: rsyncd.secrets


Herschel: PACS/RsyncAdmin (last edited 2009-07-15 14:32:36 by localhost)