Laughable performance with small files

So, I’ve got not only one, but two shiny new Drobo FS’s that I am migrating a whole host of usb drives to. For big files it’s alright, getting mostly 20-30, peaking at a steady 32 MB/s when transfering over. It’s not really fast, and nowhere near gb speeds, but it’s fast enough.

However, for small files, I am starting to think that there must be something seriously wrong with the drobos as I am only getting 100-500 KB/s. At this rate, the transfer is going to take weeks! These are files just below the 1 MB mark, and there are lots and lots (they are zipped disk images for emulators, if it makes a difference to know). Is this a joke? I know the drobo is doing its magic with calculating hashes and distributing the data across my disks (there are five of them) and whatnot, but still…

So … does anyone have the same experience, and any ideas how to improve the transfer rate for small files? I’ve tried enabling jumbo frames both on the pc doing the transfering and the Drobo, but it didn’t make any difference. My network is really simple, it’s just the PC and a switch and two drobos.

Edit: I just installed dropbear and ssh’ed in. Top tells me smbd is using just shy of 100% of CPU. Is NFS better? FTP?

Best regards,

hi one thing i noticed in general was that things speeded up if i disabled antivirus etc tools
(they have a tendancy to scan every file read and every file write depending on your settings (+ heuristics etc if exe)

if you are just copying files, and trust your existing data you could try that to see if it improves?

(also if your “source” drive is badly fragmented, it could afffect things… maybe you can try some tests (without caching) from those to the c drive or similar to see?

Not sure if any other protocol will be better - all I can say is try them and report back. Small files tend to be the worst-case scenario for any I/O as there is a fair amount of overhead with writing a file - between allocating space, updating the directory, etc. For large files this is insignificant, and almost all the time is spent copying data. For lots of tiny files, it adds up fast. I don’t think the Drobo performance should degrade that much, but the fact that it’s pegged at 100% CPU says to me that either their version of samba is poorly configured, or the DroboFS just has a shitty processor. Given that gigahertz ARM processors were shipping in phones at the time the DroboFS was released, I’m betting on the latter, and just lumping this in with cheap fans and power supplies as something DRI cut corners on.

Hello again

I installed pureftpd and transfered all the small files in no time at all (compared to the samba estimates anyway) with FileZilla and five concurrent uploads. The uploads would stall a little bit from time to time, but were mostly swiftly. I estimate the totalt transfer rate at 20-25 MB/s (cf. to SMB running at ~200 KB/s). The pureftpd processes on the drobo consumed 50-60% CPU. Thanks a bunch to whoever got pureftpd running!

Right now I’m copying some large files again, the throughput is steady at 32 MB/s, and cpu use for the smbd on the drobo is around the 60-70% mark (this is exactly the same as before enabling “jumbo frames” with MTU 9000 on both the Drobo and the windows PC doing the transfering, btw, so JF doesn’t seem to do anything for me).

Something to note with regards to pureftpd/filezilla is that some filenames with “foreign” i.e. accented characters were mangled, and ascii files were converted to unix format in the transfer. The latter could probably be turned off in the filezilla configuration. Also, a couple of zip files which had names starting with several “.” where transfered incorrectly.

I caught these mistakes by running WinMerge (free download from http://winmerge.org/) and compare the source and target folders after transfering (compare on filesize only to catch ascii converted files).

So, in my conclusion, the drobo is fine for large files, and small files if you don’t have a lot of them to copy over. If you’re doing lots and lots, say when you are migrating data to a new drobo, use FTP for that, but be aware that ftp might not handle ascii files and filenames with unusual characters as you expect. WinMerge will help alleviate this.

thanks for results post supertusse

was there a particular reason you needed to use the FTP protocol?
eg could you have mapped the usb drives and simply done a copy and paste in windows?
(that should also have made sure your files were transferred as is. none of this ascii / binary conversion nonsense to worry about) :slight_smile:

Paul, copy and paste over…an SMB share? Can you be more specific what you mean?

The SMB protocol is famously slow at copying small files. Not only that, but I can copy a lot of files and watch smbd eat all CPU on a 1GHz PIII system. Haven’t seen such problems with NFS, but I also haven’t compared the two.

Thanks for posting such detailed results, supertusse. I’m glad you found a solution, and I’m sure you’ll help many others here too.