4-bay Gen3 Ext3 Endless Drive Chatter

Drobo - End of Sale GEN 3
1

I’ve got a brand-new Drobo Gen 3 4-bay (the one that just started shipping a few weeks ago), and I’ve got four 1.5TB Seagate Barracuda drives installed in it. I bought it to use with a Linux machine, so I formatted it as one big 16TB Ext3 partition, starting on block 2048, using gdisk (gptfdisk). I mounted it on one of my computers and copied over just over 3 TB of data (via USB 2; took over a day), so it’s at 79% full.

Then I moved it to the machine it will be working with, an Intel NUC (Haswell i5 chipset) running a 3.12 Linux kernel. All the USB ports are 3.0. It mounts just fine, and from the computer’s point of view it’s just fine.

But it runs the hard drives. All the time. Constant drive head seek chatter. After several hours, the drives are quite hot. (the Drobo’s fan seems to be fine, if at low speed.) Even though nothing was accessing it, it won’t go into spin-down mode (dim LEDs). Which is odd because it did go into spin-down mode after it finished that initial 3+ TB copy.

Anyway, not being interested in losing the drives early due to wear, I shut it down cleanly. I’ve tried various tests with it; the most telling is that it sat there and ran the drives for over two days straight, never spinning down the drives, without its volume being mounted at all. (Unplugging the USB cable results in the Drobo going into standby mode, no LEDs on, within a couple of seconds.)

I’ve moved it to another machine (AMD chipset) and it seems to behave the same whether on a USB3 or USB2 port. Chatter starts within a few minutes, drives are hot within ten minutes.

Drobo Dashboard (when I have it plugged into a Mac) is convinced everything is A-OK.

So I’m wondering if anyone has any ideas of things I can try next. Thanks for your time.

2

You say you formatted it as Ext3. What tool did you use?

Did you use Dashboard to create the volumes/partitions and gdisk to format? If so, you can’t specify a location on the ‘disk’ since the OS has no access to the actual device.

You would be better served to create/format with Dashboard (NTFS for Linux), then use gdisk to just reformat as Ext3.

http://support.drobo.com/app/answers/detail/a_id/165/~/how-do-i-use-my-drobo-with-a-linux-machine%3F

3

[quote=“mgriffin34, post:2, topic:139675”]
You say you formatted it as Ext3. What tool did you use?[/quote]
gdisk first to create the partition map (GPT with protective MBR), then mke2fs -j to create the Ext3 partition.

No, I just used Dashboard to make sure it was running OK after powering up the first time and that it had the latest firmware (had to upgrade to Dashboard 2.6.0 just to see the Drobo at all).

I’ll keep that in mind if I can’t deal with this some other way than a complete erase. BTW, Dashboard only gave me the option of HFS+ on the Mac.

I read that before I even started.

  1. this isn’t a Drobo S
  2. My kernel version is 3.10. Tried both 64-bit Haswell and 64-bit AMD.
  3. GPT is supported by my tools
  4. Using Ext3 not Ext4. I’m even using mount -t ext3
  5. no idea what it means by “create the 1 TB, 2 TB volume”. Dashboard created one 16 TB “drive”. (Sometimes people use “volume” to refer to a LUN, and sometimes to a partition; not sure which is used here)
  6. Not using an Elite and not sure what “the management LUN” is
  7. Using USB3 (no FireWire interface on the Drobo or computer)
4

The most important tip is never use a tool on a Drobo that thinks it’s accessing the ‘drive’ directly.

Never touch the partition table with any tool other than Dashboard.

A HFS format would be fine. The FS format is irrelevant, you just want the partition table established. Format the volumes with whatever FS you would like.

5

Amen.

I’m not 100% positive of that. BeyondRAID is actually a “filesystem-aware” RAID setup (at least according to the patent), so you may or may not be able to go willy-nilly on it.

6

So untrue! Of course it IS relevant - Drobo has to be able to read and understand the volume bitmap to keep track of the allocation units used and assign/free/remap them dynamically (not to mention the LED capacity bar).

7

… which is why Linux users are told to only use Ext3, I guess.

So it sounds like I’m going to have to wipe the array down. Nuts.

8

OK, so I hooked the Drobo up to a Windows box, installed Dashboard 2.6.0, and factory reset the whole thing. Then I formatted it as NTFS. Then I rebooted the machine to Linux and used gdisk (a.k.a. gptfdisk) to repartition. The system hung for a couple minutes, then the drive shut itself down. From then on it wouldn’t fully boot.

So I dragged it over to my Mac, plugged it in, ejected all but two drives, and turned it back on. It reconfigured itself and came up successfully. So I used Dashboard 2.6.0 on the Mac to factory reset it, and then format as HFS+. Plugged in the other two drives. 4 TB available, so far so good.

Moved it back to Linux and repartitioned it, making sure to use the same start & end block numbers as the original partitions. This time it didn’t lock up; just repartitioned normally. So I ran mkfs.ext3 on it to create the volume (took hours), mounted it, and rsynced my 3.5 TB of data back to it. Then when done, I unmounted it but left it plugged in to the USB port. No problems indicated on the computer’s end.

That got done six hours ago. The drives are still chattering. The unit has not gone to sleep. If it’s some magic involving the partition map, it must be completely undocumented.

Bear in mind that the whole time, the storage-used LED bar has shown the correct level. The Drobo seems top be perfectly aware of the amount of space used. Also, the transfer rate during the rsync was about 30 MB/s. I don’t know how much of that is the fact that it’s on a USB 2.0 bus.

BTW, here’s the current partition map:

[code]GPT fdisk (gdisk) version 0.8.10

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sdg: 34359738368 sectors, 16.0 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): DA0E5F97-E498-44E7-9AE4-68D105A31801
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 34359738334
Partitions will be aligned on 8-sector boundaries
Total free space is 262157 sectors (128.0 MiB)

Number Start (sector) End (sector) Size Code Name
1 40 34359476183 16.0 TiB 8300 Linux filesystem
[/code]

And here’s the output from tune2fs on the one volume:

tune2fs 1.42.7 (21-Jan-2013) Filesystem volume name: drobo Last mounted on: <not available> Filesystem UUID: 079f576f-eb92-45ed-a3d3-f5a12e73ed73 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype sparse_super large_file Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 536866816 Block count: 4294934518 Reserved block count: 214746725 Free blocks: 3402898922 Free inodes: 536660513 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 4096 Inode blocks per group: 256 Filesystem created: Sat Jul 5 09:00:06 2014 Last mount time: Sat Jul 5 11:29:47 2014 Last write time: Sun Jul 6 20:13:12 2014 Mount count: 1 Maximum mount count: -1 Last checked: Sat Jul 5 09:00:06 2014 Check interval: 0 (<none>) Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 8fa9271f-d09b-402c-8c8c-01c275823b4b Journal backup: inode blocks

9

Nothing about how the Drobo interacts with the partition map is documented - it’s a proprietary system. Best bet - format it on a supported system the supported way and then stop messing with the partition table. The way you’re headed, drive chatter will be the least of your problems.

10

From the page linked by mgriffin34 above:

Please pardon my naivete.

Drobo support just told me that Drobo Gen 3 doesn’t support Linux at all (and, indeed the product page lists only NTFS and HFS+), so hopefully they’ll update their support page to reflect this.

I noticed something in your signature: “FS/EXT3 diskpack” What is that?

11

This.
As far as the chatter is concerned, Drobo does some magic optimization on the volume, even if it’s blank - don’t know if it does performance tests, or what, it’s magic…

It should go to sleep when it’s done.

Remember that Drobo is designed for consumers who have little/no knowledge of RAID and very little knowledge of filesystems, etc.

Since you’re a Linux person, you are even more prone to overthinking things with Drobo - so “play dumb” more. :wink:

12

I didn’t mean to imply ANY FS could be used on a Drobo. I was trying to state that it’s far more important to use Dashboard to create the volumes than to use Dashboard to format the filesystem.

Drobo support NTFS, HFS+ and Ext3. What tool you use to create one of those FS’s is irrelevant once the partition table is created.

I wasn’t saying you could use ZFS or btrfs on Drobo. (Might be worth noting they aren’t Filesystems and would be exempt).

BTW - I have had a Gen2 Drobo running a FAT32 filesystem with no issues for years.

13

Thanks for the encouragement, but I’ve been trying to “play dumb” from the outset. Since there’s no Linux Dashboard, there’s only so much “dumb” one can play.

Bear in mind that it behaves as if everything is working perfectly. I can mount it, unmount it, copy data to and from it, nothing gets corrupted, etc.

It just does not shut up ever.

I can’t play any dumber without simply accepting that the drives will always chatter, never spin down, and would not likely last the year.

[quote=“mgriffin34, post:12, topic:139675”]
I didn’t mean to imply ANY FS could be used on a Drobo.[/quote]
Nor did I take it as such. As noted in my tune2fs dump, I was using Ext3 – the precise filesystem specified in the “using Drobo with Linux” page you linked.

And Dashboard only allows NTFS if running on Windows, or HFS+ if running on Mac.

Not entirely correct. Drobo Support has now told me directly that Gen3 Drobo only supports NTFS and HFS+, as stated on the product page for the Gen3 4 bay.

At no point did I attempt to do so. Again, I posted the partition map dump and the tune2fs output. Single partition. Ext3. Nothing fancy. I’m not trying to format the individual drives. I’m not even using LVM on it. I’m not doing anything beyond treating it like a standard block device.

I’ll believe it. FAT32 is well documented and relatively easy to support. It does nothing weird.

Of course, it’s also not on the approved list for Gen3.

OK, just to be clear, in case it’s not: I’m not hacking the Drobo. I’m not logging into or running anything on its internal processor. I’m being far less invasive than when others install SSH on their 5N and then use that to install a bunch of other stuff. I’m doing none of that. I don’t care how they store data on the individual drives, or if they even have partition tables. I’m treating it as a Big Dumb Disk.

I sent Drobo Support their encrypted log dumps, both right after power-on and after it started chattering. And, per Drobo Support, Linux is Simply Not Supported. Not just the Dashboard. At all. They tell me that my only choices are NTFS or HFS+, neither of which have 100% reliable Linux write-mode drivers. IMHO they probably just couldn’t spare the engineering/QA resources on Ext3 support for Gen3.

I bought it because I like how my 5N handles things like failed drives and capacity upgrades. It’s a lot easier than messing with mdadm and lvm. So I figured I could just use it like the “Using with Linux” page said. My mistake, apparently.

Thank you for your time.

14

Three days straight is a long time, prior 4-bay models didn’t do data scrubbing like the 5-bays, not sure if the Gen 3 4-bay does or not.
It’s possible it’s a Seagate thing - have you checked to see if there’s an updated firmware? Drives are so complex these days… :\

As for Linux (lack of) support…
There’s drobo-utils and drobo-utils-gui though I don’t know if they’re still being maintained.

There’s a cute reference to “droboview” in the KB article about Linux usage

15

hi greyhare,
i was just wondering something… 2 things actually… :slight_smile:

  1. apart from the no linux support, did the support teams actually look at the drives themselves, in case there is a drive having some problems? the auto corrections between blocks and drives might be causing the chatter in that way.

  2. if you have the time and capability/backups etc, you could try resetting and reformatting your drobo as a windows filesystem such as ntfs, and put the same data back, and then see if you still get chatter… if you dont then repeat back to linux, and see if you get the chatter again… then you’ll know if its linux or the drives.

(you could also replace the unit but that might be overkill)

16

The drives work fine, no chatter, when the Drobo is formatted NTFS or HFS+. (Yeah, it’s annoying when “let it sit there for hours and see what happens” is part of the debugging process.) The drives themselves have been used in another (mdadm) RAID for a couple years. SMART dumps (performed pre-Drobo) show no complaints, nor the usual signs of age (uncorrectable sectors and such).

[quote=“bhiga, post:14, topic:139675”]As for Linux (lack of) support…
There’s drobo-utils and drobo-utils-gui though I don’t know if they’re still being maintained.[/quote]
drobo-utils hasn’t been updated for years. Further, Drobo is very clear that they won’t endure or support those tools (makes sense). I used the “read status” function on drobo-utils during my first attempt (it reported the correct array-fill level and no errors), and didn’t touch it on my second attempt (it would be an unknown risk).

[quote=“bhiga, post:14, topic:139675”]There’s a cute reference to “droboview” in the KB article about Linux usage
[/quote]
Yeah, I saw that. I haven’t found droboview; likely it was replaced with drobo-utils. (And it probably made some assumptions based on how the old DroboFS worked.)

Drives don’t do that for days on end unless they’re about to die. That’s enough time to scrub the whole disk several times over.

As for the support team, they asked me to hook it up to my Mac, fire up Dashboard, generate a diagnostic dump, and send it to them. I did that, and sent them another dump half an hour later when they started chattering again. Their next response, a day later, was basically “We don’t support Linux at all, only NTFS and HFS+; please refer to the Gen3 4-bay web page.”

The problem being that the computer the data’s on right now is Linux and it doesn’t have 100% guaranteed NTFS write-mode drivers. So there’s no way to tell the difference between “Drobo is chattering on its own” or “Drobo is impeded by a bug in the Linux NTFS driver.”

Yeah, I think it’s overkill. After all, it works fine with NTFS or HFS+.

17

Hey, greyhare.

I just wanted to clarify, that I agree with your methodology so far.

The only thing I was trying to point out was to use Dashboard to create the partition table. Let it format the drives as NTFS or HFS+, then use the tool of your choice to re-format (not repartition) the volumes.

Dashboard understands and recognizes more filesystems than it lets you create. I have friends who use a Drobo Mini formatted as NTFS that they move back and forth between Mac/PC all the time.

As for the scrubbing you are hearing from the drives, who knows? Drobo is very vague about BeyondRAID in every aspect, so it could be perfectly normal behavior. Who knows? Perhaps there is a BeyondRAID trigger for that model Seagate that requires a more rigorous disk test.

18

hmm, its interesting indeed, and i guess having linux without 100% ntfs support, and drobo without 100% linux support doesnt help things :slight_smile:

reading the recent posts from bhiga and mgriffin, did you actually check if there were any firmware updates for that particular model?

also, (if i dare ask), do you have the model number/firmware numbers of those drives you are using? (hides from zbig) :smiley:

one thing i can think of… (if you’re comfortable with the idea and have backups etc), is to connect the drobo to another operating system, such as mac or windows, which is not able to mount the volumes, but to keep the usb cable connected. this way, it might keep the drobo running, (eg so that it doesnt go to stand by very quickly), and still give you time to see if it starts chattering for a long time.

ive recently heard my Drobo-S chattering a lot (when its been powered up but not used for a while, and its probably scrubbing data), as well as the usual after-effects of when i copy more data to it, (where it optimises the data etc), so “some” chatter is expected, but certainly not all the time.

19

Ah, that. The FS there refers to the older “Drobo FS” NAS that preceded the 5N. It’s a fairly standard embedded Linux NAS, just with Drobo BeyondRAID storage backend (and the implementation method is very, very unique). The Drobo FS used the Ext3 filesystem, while the Drobo 5N uses Ext4. However, the %n supports migrating an FS disk pack with some minor performance loss; that’s all my sig is indicating - that I’m using an older disk pack in my 5N.

20

Ah, but there’s the rub: the volume type is recorded in the partition map. You have to change it from whatever it is to 8300 (Linux partition) so that Drobo (or anything you plug it into, really) doesn’t expect to find NTFS on the partition. To do that, you have to use a “repartitioning” tool, which updates the partition map, even if (like I did the second time) you don’t change the start or end block numbers. It’s a catch-22.

To be fair, it’s a legacy thing that goes way back before Drobo. Apparently in the old days they didn’t expect all partitions to have magic values and such so you could tell them apart on their own. You can do that these days with modern partitions, but it’s not 100% reliable, so we still have those partition types in the partition maps. (IIRC, the GPT partition map even expands these partition type labels into GUIDs.)

Dashboard 2.6.0 (the latest version, and the minimum version for Gen3) shows my partition as “Unknown” and claims the Drobo is running fine.

From what I understand, Mac OS X can reliably read and write NTFS. Apple got access to the proprietary specs for NTFS in a cross-licensing agreement years ago. Linux developers have no such access.

[quote=“mgriffin34, post:17, topic:139675”]As for the scrubbing you are hearing from the drives, who knows? Drobo is very vague about BeyondRAID in every aspect, so it could be perfectly normal behavior. Who knows? Perhaps there is a BeyondRAID trigger for that model Seagate that requires a more rigorous disk test.
[/quote]
Three. Days. Straight. Before I gave up because the drives were almost burning to the touch. That’s a sign of a runaway algorithm. Drobo Support also agreed It Should Not Do That right before playing the “we don’t support that” card.

First thing I checked. Dashboard 2.6.0 (latest) reported firmware version 3.1.0 (installed) as the latest version.

Bear in mind the Gen3 only started shipping last month. This is probably one of the first ones.

Seagate ST31500541AS (1.5 TB Barracuda LP from a couple years back). Not sure what the firmware version is; the drives are no longer installed. Bear in mind that:

[list][]The drives used to be part of a Linux/mdadm/lvm2/ext3 array and never chattered like this unless they were actually in use.
[]The drives don’t chatter if the Drobo is formatted as NTFS or HFS+.
[/list]

Under those circumstances, trying to blame the drive firmware is grasping at straws.

Did that Tuesday. It started chattering within a few minutes. Hooking up to my Mac is how I generated the diagnostic dump I sent to Drobo Support. (Which they followed up with “we don’t support Ext3 at all” a day later.)

[quote=“Paul, post:18, topic:139675”]ive recently heard my Drobo-S chattering a lot (when its been powered up but not used for a while, and its probably scrubbing data), as well as the usual after-effects of when i copy more data to it, (where it optimises the data etc), so “some” chatter is expected, but certainly not all the time.
[/quote]
Yeah, it doesn’t take over three days to scrub four 1.5 TB drives in parallel.