Pushing Files from Windows to Linux/Unix Hosts with cwRsync

“Use the best tool for the job” is a great principle. I however reserve the right to define which tool is the best when the person doing the job is going to be me. That is why I develop my Web properties, such as this blog, on a Windows PC, as I am more comfortable with Windows as a desktop platform, but for a very similar reason I run them on a Linux VPS.

This in particular means I need to deploy from Windows to Linux. Back then, I manually copied the new and changed files using the WinSCP plugin for FAR, and that was okay while there were just a few files. I also have a staging environment — a VirtualBox VM that more or less replicates my VPS setup, so I could have set up some shared folders on that VM, and then use rsync to push changes from staging to production. But instead I have set up rsync to push files right from Windows to either staging or production. Here is how you can do that too:

Setting Up cwRsync

1. Download and install the last free version of cwRsync.
2. If you are on a 64-bit system, open the cwrsync.cmd template script found in the root of your cwRsync installation in a text editor and change

   SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC

   to

   SET CWRSYNCHOME=%PROGRAMFILES(x86)%\CWRSYNC

Setting Up the Remote Host

To secure your rsync transfers, you’ll need to run them over SSH, but I presume that you don’t want to type a password each time. With public key authentication, you may create a separate passphrase-less key and restrict its use to rsync as follows:

1. On your Windows system, create a new SSH key pair without a passphrase.

   Using ssh-keygen (there is a copy in the bin\ directory of your cwRsync installation):

   ssh-keygen.exe -N "" -f key-name -C "comment"

   This will create files key-name.pub and key-name in the current directory (see the note below). These files contain public and private SSH keys respectively, the latter also known as identification. For instance:

   D:\> %homedrive%
   C:\> cd %homepath%
   C:\Users\Me\> if not exist .ssh mkdir .ssh
   C:\Users\Me\> cd .ssh
   C:\Users\Me\.ssh> "C:\Program Files (x86)\cwRsync\bin\ssh-keygen.exe" -N "" -f rsync-only -C "rsync-only"
   Generating public/private rsa key pair.
   Your identification has been saved in rsync-only.
   Your public key has been saved in rsync-only.pub.
   The key fingerprint is:
      .  .  .

   Note: The default location for OpenSSH files such as keys is the
   %HOME%\.ssh subdirectory of the user’s home directory. On Linux/Unix,
   the HOME environment variable normally points to that directory.
   The cwrsync.cmd template script therefore defines HOME as

   SET HOME=%HOMEDRIVE%%HOMEPATH%

   which translates to C:\Users\my Windows username on my Windows 7 box.

   But if you have the HOME environment variable already set, most likely by CygWin or some other Windows-Unix compatibility/interoperability package, you may wish to comment out that line from cwrsync.cmd.
   

2. Log in to your Linux box as the user that will be pushing files. That must be an unprivileged account with write permissions for the target directory(ies.)

   If that user cannot log in remotely, log in using an administrative account and use sudo to run an interactive shell as that user:

   sudo -s -H -u user

3. If the user account in question does not yet have public key authentication configured, create a directory called .ssh in its home directory, copy the public key into ~/.ssh/authorized_keys and restrict permissions:

   chmod -R go= ~/.ssh

   Otherwise, append the public key to the existing ~/.ssh/authorized_keys file.

4. As there is no passphrase on the private key, anyone who manages to take hold of it can now do an SSH login to your site. To restrict what can be done using that key, use the rrsync Perl script. It is normally installed in /usr/local/bin and symlinked to /usr/bin. On my servers, it was already present in /usr/share/doc/rsync/scripts/ in gzipped form, so I installed it as follows:

   sudo cp /usr/share/doc/rsync/scripts/rrsync.gz /usr/local/bin
   sudo gzip -d /usr/local/bin/rrsync.gz
   sudo chmod a+x /usr/local/bin/rrsync
   sudo ln -s /usr/local/bin/rrsync /usr/bin
   

   You may also download rrsync from the official rsync site.

   Now edit the authorized_keys file and insert the following before the “rsync-only” key:

   command="/usr/local/bin/rrsync subdir" ssh-rsa ... 

   where subdir is the subdirectory that will appear as root to rsync clients connecting using this key. For instance:

   command="/usr/local/bin/rrsync /srv/www" ssh-rsa ... rsync-only

   You may further restrict access to a few known hosts and/or subnets as follows:

   from="patterns",command="/usr/local/bin/rrsync subdir" ssh-rsa ...

   where patterns is a comma-separated list of host names and IP addresses that are permitted to authenticate using this key, with “?” and “*” serving as wildcards:

   from="192.168.0.*,myotherhost.com",command="/usr/local/bin/rrsync /srv/www" ssh-rsa ... rsync-only

   Note: The more obscure way of restricting access is via a Match User block in /etc/ssh/sshd_config

Customizing the cwRsync script

Back on the Windows system, copy the cwrsync.cmd template script to the desired location and append your custom rsync command to the copy:

rsync options source destination

Options

rsync has dozens of interdependent options (refer to the official man page on samba.org for details), but the two essential ones are:

-e "ssh -i private-key" or
-rsh "ssh -i private-key"

forces all communications to be conducted via the specified remote shell.

private-key is the full path to the private key in CygWin notation:

-e "ssh -i /cygdrive/c/Users/me/.ssh/rsync-only"

--chmod=permissions

Windows file/directory permissions do not quite map onto Unix, so it is necessary to explicitly specify the desired permissions. For example:

--chmod=Dug=rwX,Fug=rw,Fug-x

grants read/write permissions to file/directory owner and group, explicitly removing the execute bit from files and setting it for directories.

Most likely, you will also want to use the following options:

-r or --recursive

to copy subdirectories recursively;

-z or --compress

to compress files and conserve bandwidth;

--delete

to delete files no longer present at source;

--exclude pattern

to exclude files matching pattern – may be specified more than once, e.g.:

--exclude ".git" --exclude "cwrsync*"

and, finally

-n or --dry-run

to test your rsync command line without actually copying anything.

Source

source is the path to the source directory, again in CygWin notation:

/cygdrive/c/Work/Wordpress/themes/MyTheme/

Note: The trailing slash can make a difference! If source ends with a directory name without a trailing slash, that directory itself will be copied to destination. If there is a trailing slash, the contents of that directory will be copied. If however source ends with the current ( . ) or parent ( .. ) directory reference, the contents of the respective directoy are are synced to destination regardless of the presence of a trailiing slash.

Destination

destination has format:

user@host:path

where user is the name of the user on the target Linux/Unix host, and path is the path relative to the directory in rrsync argument (if any), in Unix notation:

wp-content/themes/zzz/

Tip: Use the caret character (^) to break long lines:

rsync.exe -riz ^
    -e "ssh -i /cygdrive/c/Users/me/.ssh/rsync-only" ^
    --delete ^
    --delete-excluded ^
    --exclude ".git" ^
    --exclude "cwrsync.*" ^
    --chmod=Dug=rwX,Fug=rw,Fug-x ^
    --log-file=cwrsync.log ^
    . rsync@myveryownhost.com:wp-content/themes/MyTheme/

That’s it. Let me know in the comments if it worked for you.

Tags: interoperability, linux, networking, rsync, windows

This entry was posted on Monday, July 16th, 2012 at 9:49 pm and is filed under Notes for myself, Small Hacks. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.