Server Infrastructure Updates - Part 2: File System Sync

I needed a way to get the code base of the sites (Mostly Drupal 7, a few wordpress, and custom coded sites), upload images and other asorted media in sync between the two servers. I looked at the couple common utilities, I've used RSync in the past to maintain backups on other servers, but it has limitations on two way sync on multiple live servers. Enter Unison, it does exactly what the name implies, unifies multiple copies of the same set of files into 2 (or more) identical copies. Another nice selling feature of unison was that it, like Rsync, works well over SSH, keeping my data safe even over the private network that my production servers share with the others in the data center. The NixCraft site has a decent article on Unison, here. Also helpful to see the user manual on UPenn's website, here.

Here is the (sanitized) code I'm using from my servers.

I'm running it as a shell script, being called every 15 minutes by a cron job. The output of the script is being piped to a log file so I included the "echo $(date)" line at the begining and end so I can track the changes being synced.

The Unison command we can break down into bits.

  • "-prefer /var/www" sets the server that is calling the unison command (in my case webserver1) as the master in the event of a conflict. Because of this, I try to make any changes to the code base on the source server (webserver1) whenever possible.
  • "-batch" batch mode: ask no questions at all. Non-conflicting changes will be propagated; conflicts will be skipped, however when coupled with the prefer flag conflicts will setled by webserver1.
  • "-sshargs '-i /root/.ssh/id_rsa'" allows me to specify any needed arguments for the SSH connection, in this case, my id_rsa file I use for passwordless SSH connections.
  • "/var/www" local sync source
  • "ssh://" the remote target

Thats pretty much it, this gives me complete file sync of all the files in my /var/www directory. I've also deployed similar scripts to sync user directories (specifically, MailDir) between servers. There are tons of other flags available for Unison, that can control all sorts of custom config settings, although I've found that the defaults work best in most situations.

Read more in the user manual.

Here are all the preferences supported by Unison. This list can be obtained by typing unison -help.

    Usage: unison [options]
        or unison root1 root2 [options]
        or unison profilename [options]

    Basic options:
     -auto              automatically accept default (nonconflicting) actions
     -batch             batch mode: ask no questions at all
     -doc xxx           show documentation ('-doc topics' lists topics)
     -fat               use appropriate options for FAT filesystems
     -group             synchronize group attributes
     -ignore xxx        add a pattern to the ignore list
     -ignorenot xxx     add a pattern to the ignorenot list
     -nocreation xxx    prevent file creations on one replica
     -nodeletion xxx    prevent file deletions on one replica
     -noupdate xxx      prevent file updates and deletions on one replica
     -owner             synchronize owner
     -path xxx          path to synchronize
     -perms n           part of the permissions which is synchronized
     -root xxx          root of a replica (should be used exactly twice)
     -silent            print nothing except error messages
     -terse             suppress status messages
     -testserver        exit immediately after the connection to the server
     -times             synchronize modification times
     -version           print version and exit

    Advanced options:
     -addprefsto xxx    file to add new prefs to
     -addversionno      add version number to name of unison on server
     -backup xxx        add a pattern to the backup list
     -backupcurr xxx    add a pattern to the backupcurr list
     -backupcurrnot xxx add a pattern to the backupcurrnot list
     -backupdir xxx     directory for storing centralized backups
     -backuploc xxx     where backups are stored ('local' or 'central')
     -backupnot xxx     add a pattern to the backupnot list
     -backupprefix xxx  prefix for the names of backup files
     -backups           keep backup copies of all files (see also 'backup')
     -backupsuffix xxx  a suffix to be added to names of backup files
     -confirmbigdel     ask about whole-replica (or path) deletes (default true)
     -confirmmerge      ask for confirmation before commiting results of a merge
     -contactquietly    suppress the 'contacting server' message during startup
     -copymax n         maximum number of simultaneous copyprog transfers
     -copyprog xxx      external program for copying large files
     -copyprogrest xxx  variant of copyprog for resuming partial transfers
     -copyquoterem xxx  add quotes to remote file name for copyprog (true/false/default)
     -copythreshold n   use copyprog on files bigger than this (if >=0, in Kb)
     -debug xxx         debug module xxx ('all' -> everything, 'verbose' -> more)
     -diff xxx          set command for showing differences between files
     -dontchmod         when set, never use the chmod system call
     -dumbtty           do not change terminal settings in text UI (default true)
     -fastcheck xxx     do fast update detection (true/false/default)
     -follow xxx        add a pattern to the follow list
     -force xxx         force changes from this replica to the other
     -forcepartial xxx  add a pattern to the forcepartial list
     -halfduplex        force half-duplex communication with the server
     -height n          height (in lines) of main window in graphical interface
     -host xxx          bind the socket to this host name in server socket mode
     -ignorearchives    ignore existing archive files
     -ignorecase xxx    identify upper/lowercase filenames (true/false/default)
     -ignoreinodenumbers ignore inode number changes when detecting updates
     -ignorelocks       ignore locks left over from previous run (dangerous!)
     -immutable xxx     add a pattern to the immutable list
     -immutablenot xxx  add a pattern to the immutablenot list
     -key xxx           define a keyboard shortcut for this profile (in some UIs)
     -killserver        kill server when done (even when using sockets)
     -label xxx         provide a descriptive string label for this profile
     -links xxx         allow the synchronization of symbolic links (true/false/default)
     -log               record actions in logfile (default true)
     -logfile xxx       logfile name
     -maxbackups n      number of backed up versions of a file
     -maxerrors n       maximum number of errors before a directory transfer is aborted
     -maxthreads n      maximum number of simultaneous file transfers
     -merge xxx         add a pattern to the merge list
     -mountpoint xxx    abort if this path does not exist
     -nocreationpartial xxx add a pattern to the nocreationpartial list
     -nodeletionpartial xxx add a pattern to the nodeletionpartial list
     -noupdatepartial xxx add a pattern to the noupdatepartial list
     -numericids        don't map uid/gid values by user/group names
     -prefer xxx        choose this replica's version for conflicting changes
     -preferpartial xxx add a pattern to the preferpartial list
     -repeat xxx        synchronize repeatedly (text interface only)
     -retry n           re-try failed synchronizations N times (text ui only)
     -rootalias xxx     register alias for canonical root names
     -rsrc xxx          synchronize resource forks (true/false/default)
     -rsync             activate the rsync transfer mode (default true)
     -selftest          run internal tests and exit
     -servercmd xxx     name of unison executable on remote server
     -showarchive       show 'true names' (for rootalias) of roots and archive
     -socket xxx        act as a server on a socket
     -sortbysize        list changed files by size, not name
     -sortfirst xxx     add a pattern to the sortfirst list
     -sortlast xxx      add a pattern to the sortlast list
     -sortnewfirst      list new before changed files
     -sshargs xxx       other arguments (if any) for remote shell command
     -sshcmd xxx        path to the ssh executable
     -stream            use a streaming protocol for transferring file contents (default true)
     -ui xxx            select UI ('text' or 'graphic'); command-line only
     -unicode xxx       assume Unicode encoding in case insensitive mode
     -xferbycopying     optimize transfers using local copies (default true)