Options Have Meanings, or, How I Made an rsync Seven Times Faster

By · Published · linux, networking

Warning: Doing this is making a clear tradeoff between security and speed. Do not do this on the public Internet or across a network you do not trust.

rsync is one of those tools that is in every computer user's toolkit. It's fantastic for moving large amounts of data around and for migrating data from one system to another.

rsync also has a ton of options and, after awhile, you get to where muscle memory means you just type the same few options over and over again. With me, that was -avz, archive, verbose, compression.

Recently, I was migrating several terabytes of data from a NAS to a computer. As is often the case, I fired up an rsync job and watched it.

It maxed out at about 35 megabit. Across a gigabit switched internal network.

I grumbled about how the NAS is slow and went to do something else, figuring this was going to take a few days to finish. But a couple hours later it occurred to me to circle back and look at this again. It was bugging me.

My first thought was to eliminate compression. This is going across an entirely internal switched network, I really don't care about reducing bandwidth utilization. So I dropped the -z options.

The result was an instant speed gain of about 5x. I was pushing 200 megabit now and the time to completion had fallen from six days to about 36 hours. Impressed, I decided to tinker a little bit more. My next idea was to eliminate any encryption (again, this is an entirely internal operation, don't do this across the Internet or on any network you don't trust). No dice, I couldn't drop encryption entirely, but I could choose the weakest cypher that both ends would support. In my case, that was aes128-ctr.

With a little bit of trial and error and some Googling, this is what I finally came up with:

rsync --info=progress2 -sav  -e "ssh -T -c aes128-ctr -o Compression=no -x" [email protected]:/path/to/copy .

No compression, weakest cypher I could find, disabled some stuff on ssh that isn't needed. With this my transfers are sustaining about 250 megabit and the time to completion has fallen to about 24 hours.

( Comments )

Did something I wrote help you out?

That's great! I don't earn any money from this site - I run no ads, sell no products and participate in no affiliate programs. I do this solely because it's fun; I enjoy writing and sharing what I learn.

All the same, if you found this article helpful and want to show your appreciation, here's my Amazon.com wishlist.

Related Posts

The Stupid Simple Guide to Setting Up Your Own DNS Server

Automatically Provisioning Polycom Phones

comments powered by Disqus