File Synchronization With Rsync(2)

File Synchronization With Rsync - Temporary Insanity
(Page 4 of 9 )

The first (and most basic) thing you can do with rsync involves using it as a more intelligent copy command on your local system. So, let's suppose that I have the following directory tree in my home area on my Linux box,

[me@olympus] $ tree -d /home/me

|-- Desktop

|   |-- Autostart

|   |-- Templates

|   `-- Trash

|-- bin

|-- mail

|-- public_html

|   |-- cache

|   `-- chatserver

|       |-- chat

|       |   |-- admin

|       |   |-- config

|       |   |-- images

|       |   |   |-- admin

|       |   |   |-- smilies

|       |   |   `-- tutorials

|       |   |-- install

|       |   |   |-- database

|       |   |   `-- languages

|       |   |-- lib

|       |   |   |-- commands

|       |   |   `-- database

|       |   `-- localization

|       |       `-- english

|       `-- docs

|-- test

`-- tmp

27 directories  

and let's further suppose that I need to copy my personal Web pages to a backup folder ("/mnt/zip/backup") on the same system.

[me@olympus] $ ls -l /mnt/zip/backup

  total 0  

With rsync, accomplishing this is a snap:

[me@olympus] $ rsync --verbose --stats --recursive public_html

  /mnt/zip/backup/Number of files: 177Number of files transferred: 158Total file size: 1043209 bytesTotal transferred file size: 1043209 bytesLiteral data: 0 bytesMatched data: 1043209 bytesFile list size: 3453Total bytes written: 15469Total bytes read: 12018  

As with a regular "cp" command, rsync needs to know the source and destination for the copy operation. Once it has this information, it compares the files in the two locations and updates the destination to an exact replica of the source. Let's see if it worked as advertised:

[me@olympus] $ ls /mnt/zip/backup/

  public_html  

Yup, it did - as you can see, my backup folder now contains a copy of my Web pages.

You can specify more than one source directory to be copied as well - let me add my "mail" directory to the list and run the command again.

[me@olympus] $ rsync --verbose --stats --recursive public_html mail

  /mnt/zip/backup/Number of files: 181Number of files transferred: 161Total file size: 1043209 bytesTotal transferred file size: 1043209 bytesLiteral data: 0 bytesMatched data: 1043209 bytesFile list size: 3513Total bytes written: 15637Total bytes read: 12066wrote 15637 bytes read 12066 bytes 18468.67 bytes/sectotal size is 1043209  speedup is 37.66  

Next, let's make a few changes to the original files, and see if rsync can detect them and selectively update the destination the next time I sync up.

[me@olympus] $ touch public_html/a.dat

  [me@olympus] $ ls > public_html/a.dat[me@olympus] $ touch public_html/b.dat[me@olympus] $ rm public_html/cache/test.html[me@olympus] $ vi public_html/error.php  

As you can see, I've added a couple of new files, deleted one old file and made changes to one PHP script. Let's sync up again and see what happens.

[me@olympus] $  rsync --verbose --stats --recursive public_html

  /mnt/zip/backup/  

Pretty cool, huh? rsync added the two extra files to my backup, and identified and copied the modified file as well. However, the single file I deleted from the source is still present in the backup - obviously, rsync didn't delete it.

A quick look at the rsync documentation clears that one up - by default, rsync doesn't delete files from the destination when synchronizing directories. This default behaviour can be overridden by adding the "--delete" parameter to the rsync command line.

[me@olympus] $ rsync --verbose --stats --recursive --delete public_html

  /mnt/zip/backup/  

And now my destination is an exact copy of my source.

[me@olympus] $ ls /mnt/zip/backup/public_html/cache/

  index.php m2_h.gif m2_n.gif tmp.gif  

It should be noted that the "--delete" option can cause substantial damage if used unwisely - the rsync manual suggests always performing a dry run first when using this option.

---

File Synchronization With Rsync - Remote Control
(Page 5 of 9 )

So that takes care of local sync - now how about remote sync?

In order to synchronize files between two hosts, it's necessary to run rsync in daemon (server) mode on one of the hosts. When running in this mode, rsync exposes a list of directories on the server; any remote host running rsync can then connect to this server and copy files to or from it.

Before you can run rsync in daemon mode, you need to configure it. This is accomplished via a configuration file, usually "/etc/rsyncd.conf" (although you can specify a different file as well, via the "--config" command-line argument). Here's an example:

 log file = /var/log/rsyncd.log

  [home]

  path = /home/me

  comment = My Home Area

  list = yes

  read only = no  

As you can see, this configuration file is similar to a standard Windows INI file, in that it is broken up into different sections or "modules", each containing variable-value pairs. Modules are identified by square braces around the module name, and lines beginning with semi-colons (;) or hashes (#) are treated as comments and ignored.

The first part of the file sets up global variables for rsync to use - in this case, it specifies the log file for rsync to use. It's also possible to specify, in this section, a welcome message that is displayed when a client attempts to connect to the server.

log file = /var/log/rsyncd.log

  motd file = /var/log/message.txt  

The second part sets up a module on the server - this is simply a directory that is available to all connecting clients. In this case, I've selected the "/home/me" directory, given it the share name "home" and set it to be writeable by all users.

[home]

   path = /home/me comment = My Home Area list = yesread only = no  

The

   path = /home/me  

option tells rsync where to locate the module on the server, while the

   list = yes  

option tells it to include the module in the list returned to connecting clients.

By default, modules on the server are not writable - that is, clients cannot upload files to the corresponding directories. This default behaviour cane be corrected via the extra

   read only = no  

option.

Once the configuration file has been saved, it's time to start up rsync in daemon mode.

[me@olympus] $ rsync --daemon  

If the rsync daemon starts up OK, you can attempt to connect to it from another host. Let's assume that this second host is named "xanadu", and already has rsync installed on it. What I'd like to do is transfer my home directory on "olympus" - the same one I backed up on the previous page - to "xanadu". Here's how I'd go about it:

[me@xanadu] $  rsync  --verbose --progress --stats --recursive

  olympus::home/ .Number of files: 230Number of files transferred: 187Total file size: 1054649 bytesTotal transferred file size: 1054649 bytesLiteral data: 1054649 bytesMatched data: 0 bytesFile list size: 4318Total bytes written: 3052Total bytes read: 1066527wrote 3052 bytes read 1066527 bytes 713052.67 bytes/sectotal size is 1054649 speedup is 0.99[me@xanadu] $ lsbin  Desktop  mail  public_html  test  tmp  

Synchronization need not be in one direction only. Using exactly the same setup as above - an rsync server on "olympus" and an rsync client on "xanadu" - it's also possible for me to transfer files in the other direction. Consider the following example, which illustrates by copying the "sql" directory to my home area on "olympus":

[me@xanadu] $ rsync --verbose --progress --stats --recursive sql
  

---

File Synchronization With Rsync - Doing More
(Page 6 of 9 )

It's possible to obtain a list of all the modules available on the rsync server by omitting the module name from the command line when connecting to the server. Here's an example, and the output:

[me@xanadu] $ rsync olympus::

  home            My Home Area  

Now, if I were to add a few more modules to the configuration file,

[temp]

   path = /tmp comment = Temp Arealist = yes  

restart the rsync server on "olympus",

[me@olympus] $ killall rsync

  [me@olympus] $ rsync --daemon  

and attempt to reconnect to it from "xanadu", I'd have access to the new modules as well.

[me@xanadu] $ rsync olympus::

  home My Home Areatemp            Temp Area  

You can exclude modules from being listed in this manner by specifying a

list = no  

option within the module configuration.

It's also possible to tell rsync to exclude certain files from the synchronization process, with the "--exclude" command-line option. Here's an example, which copies all the files *except* those with a ".tmp" extension from "xanadu" to "olympus":

[me@xanadu] $ rsync --verbose --progress --stats --recursive

  --exclude="*.tmp" olympus::home/ .  

Finally, rsync's default behaviour when encountering symbolic links is to omit them - as in the following example:

[me@xanadu] $ rsync --progress  --recursive  olympus::home/ .          

  skipping non-regular file "public_html/config.lib.php3" skippingnon-regular file "public_html/start.php"  

As you can see, when I attempt to copy the "/home/me/public_html" directory to "xanadu", every symbolic link within that directory is skipped. You can have rsync retain these links as is during the copy process by specifying the "--links" option on the command line,

[me@xanadu] $ rsync --progress  --recursive  --links olympus::home/ .  

or replace the symbolic links by the actual files being referenced with the "--copy-links" option.

[me@xanadu] $ rsync --progress  --recursive  --copy-links olympus::home/
  

---

Ref::
http://www.devshed.com/c/a/Administration/File-Synchronization-With-Rsync/3/
http://www.devshed.com/c/a/Administration/File-Synchronization-With-Rsync/4/
http://www.devshed.com/c/a/Administration/File-Synchronization-With-Rsync/5/

.