DroboFS Filesystem Benchmark (iozone 3.73)

As a follow up to rapier1’s excellent network benchmarking for the DroboFS, I decided to put the DroboFS file system through an intensive benchmark.

The benchmark is called iozone, and has become pretty much the reference for testing and evaluating storage sub-systems both in academia and in the industry.

What is iozone?

Iozone is a benchmark that simply reads and writes to a filesystem (local or remote, over NFS) and measures the throughput of those operations. The good thing about it is that it can automatically collect data for files of different sizes and different block sizes (a block being how much data is being read or written at once). On top of that, iozone can simulate almost any file access pattern imaginable, which is really good to test if a device is good enough for a specific type of application (databases, for example, have different file access patterns than media servers).

Because of that, the problem with iozone is the multitude of options. Therefore, before I show the results and my evaluation, let me go through the options I chose. The command-line I used is:

# ./iozone -a -z -i 0 -i 1 -b drobofs.xls -f /mnt/DroboFS/Shares/iozone.tmp -g 2g -R

[]-a = Used to select full automatic mode. Produces output that covers all tested file operations for record sizes of 4k to 16M for file sizes of 64k to 512M.
]-z = Used in conjunction with -a to test all possible record sizes. Normally Iozone omits testing of small record sizes for very large files when used in full automatic mode. This option forces Iozone to include the small record sizes in the automatic tests also.
[]-i 0 -i 1 = Used to specify which tests to run. (0=write/rewrite, 1=read/re-read). One will always need to specify 0 so that any of the following tests will have a file to measure.
]-b filename = Used to specify a filename that will be used for output of an Excel compatible file that contains the results.
[]-f filename = Used to specify the filename for the temporary file under test. This is where iozone performs the test.
]-g 2g = Overrides the maximum file size for auto mode to 2 GB instead of 512 MB.
-R = Generate Excel report. Iozone will generate an Excel compatible report to standard out. Just for easier reading, not really necessary.

How to run this?

Make sure that you have GNU Screen on your FS. The benchmark takes a few hours to complete, and you really don’t want an SSH session timeout to kill it before it ends. So SSH into the FS, start screen and run iozone from there.

Also, it goes without saying that you should stop all other activity on the FS, under pain of severely messing up the results. The FS, as shown by rapier1, has trouble enough keeping up with the network activity so you really don’t want skew your results because of network activity.

It would be advisable to stop DroboApps that generate background activity (such as media server apps that index the content of the Drobo periodically), but I haven’t done that, and as you’ll see, it had little impact on the overall test.

Testing environment

This is my DroboFS. I have SSH access to it, and I have compiled a lot of stuff to it, but I have no background processes besides Apache (for DroboAdmin) and uNFSd. So, no Fuppes, Firefly, or other filesystem indexing apps.

My FS is at around 75% capacity, and thus should be well into the so called “trouble zone” for Drobos, which are rumored to lose throughput as they have less free space.

The disk pack is a mish-mash of disks: 3x 500 GB salvaged from different USB cases, 1x 1.5 TB also from an external USB drive and a single new 1x 1.5 TB. I can’t remember the specific characteristics of each (in particular cache size). I guess you could say it doesn’t get worse than this. :slight_smile:

EDIT: Here is some more info. The Drobo FS has 2.11 TB out of 2.66 TB used, and firmware version is 1.0.5. (As soon as I update the firmware, I’ll rerun the test to see if there was any improvement).

What are the results?

As I said, expect this to run for a few hours (I haven’t actually an exact number, but I think mine run for 4 hours).

From the .xls file I generated two graphs: the reader and writer report. You can check the .xls for the re-reader and re-rewriter report data, but I think those are less interesting than the simple read and write tests (re-reads and re-writes usually are cached, so not much testing of the storage sub-system there…).

Here they are:
[]DroboFS Writer Report
]DroboFS Reader Report

Before I go into my analysis of these, I need to mention that although I haven’t recorded CPU usage, I have kept an eye on it using ‘top’ and my overall impression is that the average CPU usage during the test has floated around 20%, with some peaks at 60%.

Now, let’s break this down by report.

Writer Report Analysis

As you can see, block size (i.e., the size of the chunk being written) matters very little overall. Throughput is quite similar for all block sizes for any given file size, with the exception of small block sizes (4 KB and to a lesser degree 8 KB) or very large block sizes (16 MB in particular). This is pretty much standard fare on all consumer-grade devices: small blocks cause too much overhead, large blocks do not fit on the device’s write cache.

However, the file size seems to have a much more significant impact on the throughput. As expected, small file sizes do not perform as well as larger file sizes due to overhead, but there is a severe drop after 8 MB (a little over 100 MBps down to 80 MBps). After that, the throughput continues to drop until it stabilizes around 40 MBps. There are a couple of weird drop points here and there, but I think those are just due to external interferences. The maximum throughput is 103 MBps (for 4 MB files) and the minimum is 40 MBps (for 512 MB files, excluding the 3 weird points at 30 MBps).

Conclusion: It seems possible that the FS could sustain 40 MBps writes even for very large files. However, it is very surprising that files bigger than 8MB have such an impact on performance. I wouldn’t blame it all on the DroboFS though: I can’t remember the size of the HDD caches, and it is quite possible that some (several) of them have only 8 MB caches, which would explain the performance drop after 8 MB. To test this hypothesis, we need someone with better drives to run this benchmark.

Reader Report Analysis

Now this one is shocking. As before, the block size does not seem to matter much (small block sizes remarks still apply). However, the file size seems to have an incredible impact on performance. For files from 64 KB to 64 MB the FS reads like a champion. Between HDD cache, OS caching and BeyondRAID magic sauce the throughput stay almost constant and around 300 MBps (with a maximum of 330 MBps for an 8 MB file). However, as soon as iozone goes to files 128 MB and larger, the throughput drops like a stone. Average is only 39 MBps (max at 51 MBps for a 2 GB file).

The reason for this drop is a well known iozone effect. As soon as the OS runs out of memory for file caching, the performance is known to drop significantly. The iozone website has several examples like that. Since the FS only has 192 MB (at least that is what ‘free’ tells me), operations on 128 MB files overflow the cache area and thus performance drops. It is also interesting to notice that there is no “CPU cache spike” (i.e., a spike on throughput due to CPU caching) as we see on bigger Intel CPUs.

Conclusion: Although it looks really bad for files larger than 128 MB, it still is around 40 MBps. So a Drobo FS should be able to sustain 40 MBps reads even in the worst case scenario. A “simple” solution to improve the performance would be to increase the RAM on the FS, but that seems pointless since the worst case scenario matches the write speeds, and that would only offset the drop, not really fix it.

What’s next?

It would be nice if other FS owners would run the same test and post their results here. This way we could have a clearer picture of the influence of HDD cache sizes and overall free space.

As soon as I figure out a way to discover the cache sizes on my disks without pulling them out I’ll update this post with the info.

The full data set and binaries can be found here: http://s15230410.onlinehome-server.info:60080/iozone-arm-3.73/

I’m in. I’ll start the tests tonight but I have to wait until my GF leaves for work as she is using the box for time machine. How did you generate the graphs?

It’ll be nice to compare the results as when I bought my drobo I found a really good deal on 2TB WD-EADS drives ($59 each) so my drobo is filled identical drives.

Are the binaries you have posted the ones I built or yours? We should try to use the same binaries in all of this testing.

So I am running these tests now - looking at the ‘idle’ percentages I’m seeing 0.0% idle. Iozone might not be sucking up a lot of cycles but something is. The percentages associated with IO are pretty high - of course that doesn’t seem to map to a process. Or am I missing something here?

Hmm… just to confirm I started the benchmark again to specifically keep an eye on idle, and on my FS the idle percentage is exactly what is not being used by iozone (and other OS processes, such as pdflush).

Really strange indeed… your FS shouldn’t have 0% idle.

Very strange indeed. I’ll try a reboot and see if that clears anything up. Have you tried running any of the iperf tests? I’m wondering if you’ll see the same CPU limitations that I saw.

Either way, it looks like I had similar results to you. You’ll find the links to images and xls file below. Did you find out anything about your drives? I have 32MB caches across the board with a 5900 RPM spindle speed.

DFS Reader
DFS Writer
DFS Results (xls)

I haven’t managed to pull them out yet, but the thing that is very clear from your graphs is the increased number of weird spots… it really seems like there is something wrong with your Drobo, either a rogue process (unlikely) or maybe one of the disks is showing early signs of failure (much more likely, since HDDs going bad usually start slowing down for reads and writes due to bad sector replacement).

Too bad we can’t have a look at the SMART counters. That might have shed some light in your case.

Well that is troubling isn’t it? :slight_smile: I was getting pretty stellar performance before. I’ll try the test again after a reboot and see what I can find out. If necessary I can pull the drives one by one and check the smart status on them.

Hmm, my bad, I missed those questions.

Surface graph in Excel, then tweak the colors a little bit.

The binary I used is the one I compiled.[hr]

I managed to get some time to shut the Drobo down and pick the drive specs. Here they are:

1x Seagate Barracuda 7200 RPM, 500 GB, 16 MB cache (ST3500630AS).
2x WD Caviar Blue 7200 RPM, 500 GB, 16 MB cache (WD5000AAKS).
1x WD Caviar Green 5400 RPM, 1.5 TB, 32 MB cache (WD15EADS).
1x Samsung EcoGreen F2 5400 RPM, 1.5 TB, 32 MB cache (HD154UI).

As you can see, it is quite the circus in there. :smiley:

Hmm, I’m missing a Hitachi and then I’ll have one of each major brand!