Dmitry Leskov
 

Pushing Files from Windows to Linux/Unix Hosts with cwRsync

“Use the best tool for the job” is a great principle. I however reserve the right to define which tool is the best when the person doing the job is going to be me. That is why I develop my Web properties, such as this blog, on a Windows PC, as I am more comfortable with Windows as a desktop platform, but for a very similar reason I run them on a Linux VPS.

This in particular means I need to deploy from Windows to Linux. Back then, I manually copied the new and changed files using the WinSCP plugin for FAR, and that was okay while there were just a few files. I also have a staging environment — a VirtualBox VM that more or less replicates my VPS setup, so I could have set up some shared folders on that VM, and then use rsync to push changes from staging to production. But instead I have set up rsync to push files right from Windows to either staging or production. Here is how you can do that too:

Setting Up cwRsync

  1. Download and install the last free version of cwRsync.
  2. If you are on a 64-bit system, open the cwrsync.cmd template script found in the root of your cwRsync installation in a text editor and change

    SET CWRSYNCHOME=%PROGRAMFILES%\CWRSYNC

    to

    SET CWRSYNCHOME=%PROGRAMFILES(x86)%\CWRSYNC

Setting Up the Remote Host

To secure your rsync transfers, you’ll need to run them over SSH, but I presume that you don’t want to type a password each time. With public key authentication, you may create a separate passphrase-less key and restrict its use to rsync as follows:

  1. On your Windows system, create a new SSH key pair without a passphrase.

    Using ssh-keygen (there is a copy in the bin\ directory of your cwRsync installation):

    ssh-keygen.exe -N "" -f key-name -C "comment"

    This will create files key-name.pub and key-name in the current directory (see the note below). These files contain public and private SSH keys respectively, the latter also known as identification. For instance:

    D:\> %homedrive%
    C:\> cd %homepath%
    C:\Users\Me\> if not exist .ssh mkdir .ssh
    C:\Users\Me\> cd .ssh
    C:\Users\Me\.ssh> "C:\Program Files (x86)\cwRsync\bin\ssh-keygen.exe" -N "" -f rsync-only -C "rsync-only"
    Generating public/private rsa key pair.
    Your identification has been saved in rsync-only.
    Your public key has been saved in rsync-only.pub.
    The key fingerprint is:
       .  .  .
    Note: The default location for OpenSSH files such as keys is the
    %HOME%\.ssh subdirectory of the user’s home directory. On Linux/Unix,
    the HOME environment variable normally points to that directory.
    The cwrsync.cmd template script therefore defines HOME as

    SET HOME=%HOMEDRIVE%%HOMEPATH%

    which translates to C:\Users\my Windows username on my Windows 7 box.

    But if you have the HOME environment variable already set, most likely by CygWin or some other Windows-Unix compatibility/interoperability package, you may wish to comment out that line from cwrsync.cmd.

  2. Log in to your Linux box as the user that will be pushing files. That must be an unprivileged account with write permissions for the target directory(ies.)

    If that user cannot log in remotely, log in using an administrative account and use sudo to run an interactive shell as that user:

    sudo -s -H -u user
  3. If the user account in question does not yet have public key authentication configured, create a directory called .ssh in its home directory, copy the public key into ~/.ssh/authorized_keys and restrict permissions:

    chmod -R go= ~/.ssh

    Otherwise, append the public key to the existing ~/.ssh/authorized_keys file.

  4. As there is no passphrase on the private key, anyone who manages to take hold of it can now do an SSH login to your site. To restrict what can be done using that key, use the rrsync Perl script. It is normally installed in /usr/local/bin and symlinked to /usr/bin. On my servers, it was already present in /usr/share/doc/rsync/scripts/ in gzipped form, so I installed it as follows:

    sudo cp /usr/share/doc/rsync/scripts/rrsync.gz /usr/local/bin
    sudo gzip -d /usr/local/bin/rrsync.gz
    sudo chmod a+x /usr/local/bin/rrsync
    sudo ln -s /usr/local/bin/rrsync /usr/bin
    

    You may also download rrsync from the official rsync site.

    Now edit the authorized_keys file and insert the following before the “rsync-only” key:

    command="/usr/local/bin/rrsync subdir" ssh-rsa ... 

    where subdir is the subdirectory that will appear as root to rsync clients connecting using this key. For instance:

    command="/usr/local/bin/rrsync /srv/www" ssh-rsa ... rsync-only

    You may further restrict access to a few known hosts and/or subnets as follows:

    from="patterns",command="/usr/local/bin/rrsync subdir" ssh-rsa ...

    where patterns is a comma-separated list of host names and IP addresses that are permitted to authenticate using this key, with “?” and “*” serving as wildcards:

    from="192.168.0.*,myotherhost.com",command="/usr/local/bin/rrsync /srv/www" ssh-rsa ... rsync-only

    Note: The more obscure way of restricting access is via a Match User block in /etc/ssh/sshd_config

Customizing the cwRsync script

Back on the Windows system, copy the cwrsync.cmd template script to the desired location and append your custom rsync command to the copy:

rsync options source destination

Options

rsync has dozens of interdependent options (refer to the official man page on samba.org for details), but the two essential ones are:

-e "ssh -i private-key" or
-rsh "ssh -i private-key"
forces all communications to be conducted via the specified remote shell.

private-key is the full path to the private key in CygWin notation:

-e "ssh -i /cygdrive/c/Users/me/.ssh/rsync-only"
--chmod=permissions
Windows file/directory permissions do not quite map onto Unix, so it is necessary to explicitly specify the desired permissions. For example:

--chmod=Dug=rwX,Fug=rw,Fug-x

grants read/write permissions to file/directory owner and group, explicitly removing the execute bit from files and setting it for directories.

Most likely, you will also want to use the following options:

-r or --recursive
to copy subdirectories recursively;
-z or --compress
to compress files and conserve bandwidth;
--delete
to delete files no longer present at source;
--exclude pattern

to exclude files matching pattern – may be specified more than once, e.g.:

--exclude ".git" --exclude "cwrsync*"

and, finally

-n or --dry-run
to test your rsync command line without actually copying anything.

Source

source is the path to the source directory, again in CygWin notation:

/cygdrive/c/Work/Wordpress/themes/MyTheme/

Note: The trailing slash can make a difference! If source ends with a directory name without a trailing slash, that directory itself will be copied to destination. If there is a trailing slash, the contents of that directory will be copied. If however source ends with the current ( . ) or parent ( .. ) directory reference, the contents of the respective directoy are are synced to destination regardless of the presence of a trailiing slash.

Destination

destination has format:

user@host:path

where user is the name of the user on the target Linux/Unix host, and path is the path relative to the directory in rrsync argument (if any), in Unix notation:

wp-content/themes/zzz/

Tip: Use the caret character (^) to break long lines:

rsync.exe -riz ^
    -e "ssh -i /cygdrive/c/Users/me/.ssh/rsync-only" ^
    --delete ^
    --delete-excluded ^
    --exclude ".git" ^
    --exclude "cwrsync.*" ^
    --chmod=Dug=rwX,Fug=rw,Fug-x ^
    --log-file=cwrsync.log ^
    . rsync@myveryownhost.com:wp-content/themes/MyTheme/

That’s it. Let me know in the comments if it worked for you.

Tags: , , , ,

« | »

Talkback

  1. lley
    25-Jun-2013
    4:15 am
    1

    “Back on the Windows system, copy the cwrsync.cmd template script to the desired location”

    what do you mean by desired location?

    tx

  2. Dmitry Leskov
    07-Jul-2013
    1:54 pm
    2

    I typically copy the cwrsync.cmd template into the parent of the folder that I need to synchronize. Another option is to place the script right in that folder and add --exclude "cwrsync.*" to the custom rsync command line.

  3. alex
    04-Oct-2014
    6:42 pm
    3

    works great, thx!!!

  4. Derek Illchuk
    03-Mar-2015
    10:11 pm
    4

    Very nice, thank you. One suggestion: don’t allow SSH to fall back to password authentication, as follows:

    -e “ssh -o PasswordAuthentication=no -i…

    Otherwise, you may wreak devastation when it defaults to, say, the user’s home directory (vs. what is specified in the authorized_keys rrsync command).

  5. Thom Mason
    17-Nov-2015
    9:39 am
    5

    When I try this I get WARNING UNPROTECTED PRIVATE KEY FILE

    Permissions 0660 for /cygdrive/c/Users/thom_000/.ssh/rsync-only are too open

    Can’t find a setting using windows permissions that fixes this

    running cwrsync only – not full cygwin

  6. Jake
    28-Apr-2016
    1:45 pm
    6

    Thom,

    I struggled with the same and wasted an hour to Google the problem. The fastest way to solve it is to:

    1) Install Cygwin. Just install the default settings (i.e. minimal package). It takes about 100MB and just few clicks. https://cygwin.com/install.html

    2) Open Cygwin (it opens a Bash console)

    3) Go to the directory where your private key lies:

    cd /cygdrive/c/Users/YOURUSERNAME/.ssh

    4) Run:

    chgrp Users PRIVATE_KEY_FILENAME
    chmod 600 PRIVATE_KEY_FILENAME

    This took some 15 minutes including downloading Cygwin. Much faster than endless Googling without a solution.

    I hope someone else struggling with the same problem will find this too!

  7. John Sullivan
    11-May-2016
    12:26 am
    7

    Thom Mason – I worked around the permission error by using File Explorer to select the properties for the private key file. In the Security tab choose advanced permissions, then change the owner to the Administrator of the local machine rather than your account. Once I made this change, I stopped getting the permission error.

  8. jack
    18-Jan-2018
    9:28 pm
    8

    use robocopy.

* Copy This Password *

* Type Or Paste Password Here *