comp.os.linux.networking

On Wed, 19 Mar 2008 11:52:11 +0000, Jack Snodgrass wrote:

> I use rsync over ssh for 99% of my stuff.

Yeah, me too. I'm mirroring about 300 remote partitions every night,
maybe 1/2 TB data in all. At midnight cron kicks off a ~500 line bash
script on 8 raid boxes and generally they finish in two hours or less.

I generated rsa keys for each of the servers and sent them to the remote
boxes they would be mirroring. There's a staging area for new partitions,
which are automatically copied to the main archive if they successfully
pull in the remote partition. Any mirrors in the main archive also have
snapshots generated going back a week.

In general about 10 rsync jobs fail per night, or less. The majority of
those are simple connect failures, where the remote VPN link is down for
one reason or another. If an rsync job starts and the connection drops,
the script recovers the mirror to the last known good state (generally
the previous day).

We first got some estimates for a commercial solution to backup these
partitions and it was just incredibly expensive. The solution we are
using was less than 20k to implement. Even if cost were not an issue it's
hard to imagine any commercial product outperforming the current setup.

Rsync has completely changed how we work.