As a follow up to rapier1’s excellent network benchmarking for the DroboFS, I decided to put the DroboFS file system through an intensive benchmark.
The benchmark is called iozone, and has become pretty much the reference for testing and evaluating storage sub-systems both in academia and in the industry.
What is iozone?
Iozone is a benchmark that simply reads and writes to a filesystem (local or remote, over NFS) and measures the throughput of those operations. The good thing about it is that it can automatically collect data for files of different sizes and different block sizes (a block being how much data is being read or written at once). On top of that, iozone can simulate almost any file access pattern imaginable, which is really good to test if a device is good enough for a specific type of application (databases, for example, have different file access patterns than media servers).
Because of that, the problem with iozone is the multitude of options. Therefore, before I show the results and my evaluation, let me go through the options I chose. The command-line I used is:
# ./iozone -a -z -i 0 -i 1 -b drobofs.xls -f /mnt/DroboFS/Shares/iozone.tmp -g 2g -R
-a = Used to select full automatic mode. Produces output that covers all tested file operations for record sizes of 4k to 16M for file sizes of 64k to 512M.
-z = Used in conjunction with -a to test all possible record sizes. Normally Iozone omits testing of small record sizes for very large files when used in full automatic mode. This option forces Iozone to include the small record sizes in the automatic tests also.
-i 0 -i 1 = Used to specify which tests to run. (0=write/rewrite, 1=read/re-read). One will always need to specify 0 so that any of the following tests will have a file to measure.
-b filename = Used to specify a filename that will be used for output of an Excel compatible file that contains the results.
-f filename = Used to specify the filename for the temporary file under test. This is where iozone performs the test.
-g 2g = Overrides the maximum file size for auto mode to 2 GB instead of 512 MB.
-R = Generate Excel report. Iozone will generate an Excel compatible report to standard out. Just for easier reading, not really necessary.
How to run this?
Make sure that you have GNU Screen on your FS. The benchmark takes a few hours to complete, and you really don’t want an SSH session timeout to kill it before it ends. So SSH into the FS, start screen and run iozone from there.
Also, it goes without saying that you should stop all other activity on the FS, under pain of severely messing up the results. The FS, as shown by rapier1, has trouble enough keeping up with the network activity so you really don’t want skew your results because of network activity.
It would be advisable to stop DroboApps that generate background activity (such as media server apps that index the content of the Drobo periodically), but I haven’t done that, and as you’ll see, it had little impact on the overall test.
This is my DroboFS. I have SSH access to it, and I have compiled a lot of stuff to it, but I have no background processes besides Apache (for DroboAdmin) and uNFSd. So, no Fuppes, Firefly, or other filesystem indexing apps.
My FS is at around 75% capacity, and thus should be well into the so called “trouble zone” for Drobos, which are rumored to lose throughput as they have less free space.
The disk pack is a mish-mash of disks: 3x 500 GB salvaged from different USB cases, 1x 1.5 TB also from an external USB drive and a single new 1x 1.5 TB. I can’t remember the specific characteristics of each (in particular cache size). I guess you could say it doesn’t get worse than this.
EDIT: Here is some more info. The Drobo FS has 2.11 TB out of 2.66 TB used, and firmware version is 1.0.5. (As soon as I update the firmware, I’ll rerun the test to see if there was any improvement).
What are the results?
As I said, expect this to run for a few hours (I haven’t actually an exact number, but I think mine run for 4 hours).
From the .xls file I generated two graphs: the reader and writer report. You can check the .xls for the re-reader and re-rewriter report data, but I think those are less interesting than the simple read and write tests (re-reads and re-writes usually are cached, so not much testing of the storage sub-system there…).
Before I go into my analysis of these, I need to mention that although I haven’t recorded CPU usage, I have kept an eye on it using ‘top’ and my overall impression is that the average CPU usage during the test has floated around 20%, with some peaks at 60%.
Now, let’s break this down by report.
Writer Report Analysis
As you can see, block size (i.e., the size of the chunk being written) matters very little overall. Throughput is quite similar for all block sizes for any given file size, with the exception of small block sizes (4 KB and to a lesser degree 8 KB) or very large block sizes (16 MB in particular). This is pretty much standard fare on all consumer-grade devices: small blocks cause too much overhead, large blocks do not fit on the device’s write cache.
However, the file size seems to have a much more significant impact on the throughput. As expected, small file sizes do not perform as well as larger file sizes due to overhead, but there is a severe drop after 8 MB (a little over 100 MBps down to 80 MBps). After that, the throughput continues to drop until it stabilizes around 40 MBps. There are a couple of weird drop points here and there, but I think those are just due to external interferences. The maximum throughput is 103 MBps (for 4 MB files) and the minimum is 40 MBps (for 512 MB files, excluding the 3 weird points at 30 MBps).
Conclusion: It seems possible that the FS could sustain 40 MBps writes even for very large files. However, it is very surprising that files bigger than 8MB have such an impact on performance. I wouldn’t blame it all on the DroboFS though: I can’t remember the size of the HDD caches, and it is quite possible that some (several) of them have only 8 MB caches, which would explain the performance drop after 8 MB. To test this hypothesis, we need someone with better drives to run this benchmark.
Reader Report Analysis
Now this one is shocking. As before, the block size does not seem to matter much (small block sizes remarks still apply). However, the file size seems to have an incredible impact on performance. For files from 64 KB to 64 MB the FS reads like a champion. Between HDD cache, OS caching and BeyondRAID magic sauce the throughput stay almost constant and around 300 MBps (with a maximum of 330 MBps for an 8 MB file). However, as soon as iozone goes to files 128 MB and larger, the throughput drops like a stone. Average is only 39 MBps (max at 51 MBps for a 2 GB file).
The reason for this drop is a well known iozone effect. As soon as the OS runs out of memory for file caching, the performance is known to drop significantly. The iozone website has several examples like that. Since the FS only has 192 MB (at least that is what ‘free’ tells me), operations on 128 MB files overflow the cache area and thus performance drops. It is also interesting to notice that there is no “CPU cache spike” (i.e., a spike on throughput due to CPU caching) as we see on bigger Intel CPUs.
Conclusion: Although it looks really bad for files larger than 128 MB, it still is around 40 MBps. So a Drobo FS should be able to sustain 40 MBps reads even in the worst case scenario. A “simple” solution to improve the performance would be to increase the RAM on the FS, but that seems pointless since the worst case scenario matches the write speeds, and that would only offset the drop, not really fix it.
It would be nice if other FS owners would run the same test and post their results here. This way we could have a clearer picture of the influence of HDD cache sizes and overall free space.
As soon as I figure out a way to discover the cache sizes on my disks without pulling them out I’ll update this post with the info.
The full data set and binaries can be found here: http://s15230410.onlinehome-server.info:60080/iozone-arm-3.73/