Performance problems

[agoz]

I’m a Mac software developer developing high performance realtime software, and after spending a week with the Drobo FS I have to conclude that it’s… aehm… “no comment”.

Drobo FS firmware 1.1.1 with 3 WD Green 2TB drives (EARS) installed, connected directly to a MacPro with OSX 10.6.6, using jumbo frames. All drives are new and in good S.M.A.R.T. shape, as checked outside of the Drobo FS.

We have seen a never-ending list of problems, but let’s start from these:

1. how can be that deleting folders takes 100% of CPU time on I/O? You can check this yourself: log as root the start a rm -rvf on a folder containing a thousand of 8MB files, and check top.

CPU: 0.0% usr 0.6% sys 0.0% nic 0.0% idle 99.4% io 0.0% irq 0.0% sirq

It simply doesn’t make sense… deleting files should require a very limited number of disk accesses. Note that this has nothing to do with network access or connection, speed… it’s all happening in the drobo fs alone.

2. apart from the 100% CPU used in reading disks, you can measure how it takes roughly one second to delete each file (you can check this by using tcpdump on the ethernet port and then deleting from the computer… it take 1 or 2 seconds for each file to be deleted). This is not a network issue. Even deleting with rm -rvf from the drobo shell, after 30 minutes it’s still using 100% CPU time on disk I/O deleting those thousand files (absurd!).

3. The Drobo FS is horrendously RAM starved, i.e. there’s only about 100MB of RAM for caching drive access, i.e. 20MB for each drive… It’s ridiculous, as modern drives have 64MB of cache each.

4. When the Drobo FS takes so much time deletin or creating files, the host computer often timeouts or locks expecting a reaction from the drive, and we have to stop and reboot machines in order to go back and use them.

5. We have tried for a week with no success to use Time Machine to create a ~ 2TB backup from a MacPro to Drobo. We used a dedicated one to one connection, no router, static IPs, Jumbo frames, SFTP Cat 6 cable. The same connection with another MacPro does about 60MB/sec average transfer speed. The process never reaches the end. The Drobo FS will sometimes stall and lock (you can check it using tcpdump, it stops responding) and we have to start again. Switching to normal frames makes no difference.

If someone at Drive Robotics wants us to test something here we are…

Andrea
Redmatica

[ricardo]

I’m not a DRI employee, but let me see if I can help you with some pointers.

1. how can be that deleting folders takes 100% of CPU time on I/O?

2. apart from the 100% CPU used in reading disks, you can measure how it takes roughly one second to delete each file (you can check this by using tcpdump on the ethernet port and then deleting from the computer… it take 1 or 2 seconds for each file to be deleted).

As far as I was able to dig into the FS guts, the architecture goes like this:

Network:    [ SMB, AFP, and NFS           ]
Filesystem: [ Ext3 with journaling        ]
Driver:     [ BeyondRAID as a SCSI driver ]
Hardware:   [ Disk pack                   ]

As you can see, the IO usage is probably due to BeyondRAID’s SCSI driver, since it runs in the kernel. However, the choice of ext3 is not helping either. Ext3 has notoriously bad performance for deletes.

3. The Drobo FS is horrendously RAM starved, i.e. there’s only about 100MB of RAM for caching drive access, i.e. 20MB for each drive… It’s ridiculous, as modern drives have 64MB of cache each.

No argument from me there. It is a real shame that it does not ship with at least 512 MB.

4. When the Drobo FS takes so much time deletin or creating files, the host computer often timeouts or locks expecting a reaction from the drive, and we have to stop and reboot machines in order to go back and use them.

Which protocol are you using? Usually this can be worked around by tweaking the timeouts on the client side.

5. We have tried for a week with no success to use Time Machine to create a ~ 2TB backup from a MacPro to Drobo. We used a dedicated one to one connection, no router, static IPs, Jumbo frames, SFTP Cat 6 cable. The same connection with another MacPro does about 60MB/sec average transfer speed. The process never reaches the end. The Drobo FS will sometimes stall and lock (you can check it using tcpdump, it stops responding) and we have to start again. Switching to normal frames makes no difference.

I’m not sure about this one. To be honest, I know someone that has the same kind of problem with TimeCapsules from Apple, so I’m not sure it is a transfer speed problem, but instead a TimeMachine problem instead (as confirmed by a lot of Google hits).