Improving rsync performance?

Drobo - End of Sale Drobo 5N
1

I’m a long time Drobo user (I have and old 2G unit that’s still ticking along), and since I needed some new consolidated storage, and reading lots of posts and reviews about how the 5N and FS were focused on speed (low performance was my biggest problem w/ the old Drobo), I figured that I’d give it a try.

I’m running a new mixed array of drives (2x3TB, 3x2TB for about 8TB of single redundancy storage). I’ve also added a 32GB SSD I had laying around in the mSATA accelerator (a MyDigitalSSD SuperCache 2 SATA 6G drive w/ a PHISON S9 controller - not stunning perf, but 200MB/s+ read/writes, and 10K+ IOPs, so still better than spinning rust I’d imagine).

I’m running two rsyncs from two computers, one from the old Drobo attached to FW800 on my Mac Pro, and another, a 4TB USB 3.0 drive attached to a Macbook Air w/ a Thunderbolt Gigabit ethernet connector. Both computers are hardwired w/ CAT6 via a Netgear router w/ GigE (a WNR3500L running Tomato).

When transferring raw files I have no problem pushing a decent (50MB/s) transfer rate. However, rsyncing is just abysmal. Here’s a sample of the iostat:

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    99.50  229.85   89.05   931.34   807.96    10.91     4.41   13.78   18.79    0.84   3.06  97.51

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00   112.44  197.01  114.43   811.94   953.23    11.34     5.59   16.77   23.51    5.17   3.16  98.51

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.50    85.57  258.21   72.64  1050.75   678.61    10.45     5.66   18.23   22.95    1.44   3.02 100.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.50    81.09  213.43   69.15   869.65   644.78    10.72     5.56   19.15   25.17    0.58   3.50  99.00

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00   105.47  153.73  104.98   722.39   883.58    12.42     3.31   13.38   19.35    4.64   3.85  99.50

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
sda               0.00    61.19  155.72   54.73   632.84   495.52    10.72     4.82   22.88   30.61    0.91   4.68  98.51

Basically it looks like it’s at 100% I/O utilization (total CPU usage is almost completely idle, load is at 3) pushing about 1MB/s while syncing.

Has anyone had better success with rsyncing w/ the 5N? Is this normal performance?

Any thoughts on how I might be able to improve the rsync performance? I don’t mind getting a better mSATA SSD if that would help significantly (I’d probably go for a 120GB Crucial M500), although I’d want to make sure that’s actually the bottleneck.

I noticed that cpuinfo says there are 3 cores - does that mean there’s only 1 core dedicated to the actual file writing and that’s what’s maxing out?

(I have about 6.5M files on the Drobo, and probably something similar on the USB drive.)

2

For rsync, what happens if you simplify the sync process, for instance with “–size-only” (if that’s a valid method for your files)? You might be getting bogged down with lots of tiny reads to compare metadata.

Based on what I was told by Micron/Crucial yesterday, don’t buy a Crucial SSD for your Drobo - they’ve listed them as incompatible and will not warranty them for use in the Drobo. Meaning when the 5N kills it in a year, they’re not going to replace it. Having had my 5N kill my Crucial M4 this week (which prompted said call), I’d be wary.

3

Based on the r_awaits and the avgrq-sz I suspect you could be right - that it’s getting bogged down w/ metadata. I’m a couple days in and having restarted before and taken days to catch up, I’m hesitant to restart again on speculation though -
I’ll probably just let this run through but will maybe give --size-only (or maybe even --ignore-existing for comparison) a try on a test set and see how it compares.

I searched the forum and was a bit surprised to not see any performance comparison/results w/ rsync, since that seems like a pretty common use case (certainly it’ll be my most common scenario for backing up files).

Thanks for the tip about Crucial’s warranty coverage. That’s a real bummer about it’s write longevity in the 5N. We have some extra MSATA M500’s in the office so I may stick one in just to test if it affects the rsync performance. If it’s a big enough boost, it may be worth putting in even as a $100/yr consumable. What happens when the mSATA accelerator dies? Does the Drobo handle this failure gracefully? I’m assuming you didn’t have any data corruption?

4

Oh, the array is fine - I just got a blinking red light in the Dashboard (although nothing on the unit as I recall) and it said the accelerator had failed and needed to be replaced. No fuss. But I can think of ways I’d rather spend that $100/year. :slight_smile:

5

I didn’t get around to doing benchmarks (still just trying to transfer files), but I can give a bit of an update. I’ve installed and am using rsyncd directly which is a bit (but not significantly) faster than rsyncing via afp mount.

I’ve replaced my previous temporary mSATA card (~10K write IOPS) w/ a much faster 128GB Plextor M5M card (rated at 76K write IOPS).

This has improved the Drobo 5N’s IOPS from about 250-300 before to about 700. svctm has decreased to <2ms, utilization pegged at 100%. This is a bit too bad as the raw spinning drives themselves should be providing 700+ IOPS in a regular system.

At the end of the day, I guess the Drobo 5N is just still plain old pokey, even for raw/regular usage. I’m also migrating data to/from the 5N and a bunch of external (FW800/USB3) drives and an old microserver (the HP N36L - 2x1.3GHz Athlon, 4GB RAM, 4xslow 5400RPM drives in an LVM), and it’s night and day how much more responsive it is. It’s multiple times faster both syncing files, and running local operations like du (much less convenient to upgrade drives, of course).

I guess I’m pretty committed to the 5N at this point since it’d take a week+ to move files off, but I’ll probably end up building my own ZFS/btrfs NAS next time I upgrade.

6

I use my Drobo 5N to archive a bunch of stuff from my server to series of directories on the Drobo using a backup script that’s been honed over the years. For directories of many “normal” sized files I use rsync but for anything where I am pushing large tar files or compressed dumps over to archive storage I use dd on a mounted Drobo 5N share. rsync on the Drobo5N is slow and I have never been able to tweak it enough to compare with a raw copy.

As you mention I get about 50MB/s moving 50GB archive files using plain copy but using rsync I get about 22MB/s. Using the -W arg helps a little as it suppresses the delta algorithm and behaves more like a straight copy but not much. rsync is great for syncing two vast file systems when it’s not feasible to know what’s changed but if I know what I need to move a plain copy is much faster.

7

If I may offer my two cents: if your raw download speed is that much faster than what rsync is able to do when running on the 5N, why not mount the 5N share on a powerful machine and perform the whole rsync “locally”?

I.e., instead of asking a remote rsync instance to calculate the deltas and so on (which probably is what is choking the 5N’s CPU), why not do it all locally and let the 5N just do what it does best (which is to serve data).

I never benchmarked this scenario since all of my rsync needs were pretty much background batch jobs, but if you need a more responsive rsync maybe that would be the way to address it?

8

Just a quick followup, at least in my case, the 5N’s 3 system CPUs aren’t the problem, they sit idle, it’s the single “drive” CPU that maxes out. Performance is actually worse w/ being locally mounted vs running via rsync as there’s extra overhead, but in the end, it’s entirely IO bound. I don’t mind running my rsyncs and background jobs, but that they take days to run… is disappointing to say the least.