How do I force a rebuild, when a drive is intentionally missing?

Forum History General Discussion
1

I’m still debugging some of the issues related to the Drobo corrupting my disk pack during a firmware upgrade, and wanted to float another idea.

When I insert disks from the disk pack that the Drobo damaged into the Drobo under Windows, it keeps prompting me to insert another disk, even beyond the number in the disk pack itself (4 bays, all 4 drives inserted, Drobo continues to prompt me to add all of the disks in the disk pack).

How do I force it to accept 3 out of the 4 disks, and rebuild/resilver the data across a blank 4th drive (or no drive at all, as if the drive controller in the 4th drive was physically damaged, for example).

I don’t see a way to do this… if I insert 3 of the 4 drives and say “Ok Drobo, that’s it… fix the data so it consumes only 3 drives now.”, I can’t.

Ideas?

2

basically you cant, if it is capable of doing it, it does so automatically, if it doesnt do it automatically then something is wrong, either it doesnt have enough data to rebuild from, or, as in your case with 4out of 4 drives, something has gone very wrong and it thinks it need more data to commence the rebuild

3

Ok, let’s go with the facts:

I have 3500GB of physical storage in my Drobo, across 4 drives (1TB/750GB/1TB/750GB), and on that array, is stored 1.2TB of physical bits and bytes. drobom reports the array as being 82% filled, but I don’t trust that, because Drobo can’t correctly calculate this anyway.

So now the firmware trashes the disk pack, so the Drobo no longer recognizes it. I pull a single drive, and boot it back up.

Why wouldn’t the Drobo say “Oops, a drive failed, I’ll protect your data, but you’ll be in degraded mode until you add another drive.”?

This is no different from a drive physically failing… (motor dies, controller fried, reach MTBF rating of drive, overheat, etc.)

The Drobo must be able to deal with a physically failed drive, right? Or is that not possible?

4

yes it can,

but since in your case it cant cope with 4 fully operational drives then evidently there is a deeper problem.

5

This may be a clue to the overall issue I’m facing…

When I plugged in 3 drives, and as the Drobo was doing the chase, I seated the 4th drive, which got all 10 LEDs lit (as suggested by someone in another post), I notice the physical device is handed to the OS, but not the partitions on it.

drobom from the drobo-utils package sees the underlying filesystem as FAT32, not ext3.

I wonder if the Drobo, during the firmware upgrade got “confused” and made the underlying filesystem identifier in the header of the actual device, FAT32. When the Drobo tries to address the filesystem as FAT32 (to check free space, etc.), the device hangs at boot, because the actual bits and bytes of the filesystem itself, are ext3, not FAT32.

Either that, or the Drobo itself is telling the host OS that the filesystem (which it hasn’t handed over to the OS yet) is FAT32, when in fact it really is ext3.

This gets more interesting the more I dig into it…

6

there was a guy in a differnet thread (posted today) who was trying to check the health of his file system and was told ext3 is not supported - i wonder if they are giving up on it? this could be a sign of that perhaps?

http://www.drobospace.com/forums/showthread.php?tid=841&pid=5858#pid5858

“. Support told me they do not provide support for EXT3”

7

Well, since the DroboShare formats volumes itself, to ext3, I would imagine they have to support it, because their device supports it, engineered into the tools itself.

But whether or not they provide user-level support is another thing entirely.

I’ve had some discussions with some embedded developer colleagues, and the concensus is that it is possible to get console output from the Drobo, at boot time.

The device is capable of sending commands to the Drobo over USB, and receiving data back that is not data destined for the disk packs themselves (“Get Diagnostics” is one example of such a pathway). So the physical device and cable are capable of sending and receiving commands not directly related to the storage of date on the volumes.

Writing a simple “console” application to view the device’s output at boot time should not be an impossible matter. If all else fails, I’ll be chasing that possibility down.

In parallel, since DRI refuses to help me further at this point, and is completely ignoring my open incidents, I will be contacting WindRiver directly, for specifications on the underlying hardware used in the device, and building a custom serial connector to attempt to get lower-level diagnostics out of the device. I have a few friends in the WindRiver community that I may be able to call in some favors on to help me.

The good news is that my results, experiences and any code/tools/scripts and diagnostics that result from this will be made publicly accessible (outside this forum) for others to take advantage of, as they migrate their data away from being stored on these devices.

The data is the most-important thing, and when a vendor device locks me out from accessing my own data, it’s time to vote with my wallet (and my voice in the online community).