Drobo

Clean disk pack, more debugging 'de blinking lights

In parallel to imaging my disk pack for offline analysis, I decided to recreate the scenario I’ve been affected by, under “supported” conditions, to validate that what I was seeing, was actually a Drobo problem, and not a filesystem or LUN size problem.

The problem I’ve been hit with, is an upgrade of my Drobo from firmware 1.3.1 to 1.3.5 has rendered the Drobo’s ability to successfully boot the disk pack, impossible.

Here’s what I did:

  1. Under Windows, with the latest supported Drobo Dashboard, I flashed the Drobo unit itself to the version 1.3.1 firmware that it was previously running. This was recommended by DRI support (case #00029596)

  2. I inserted 4x1TB SATA disks, all absolutely identical in model, revision and speed, and created a single 2TB LUN out of them

  3. I shut it down and powered it on several times to verify that the 1.3.1 x 2TB LUN disk pack functions as expected under Windows and my preferred host; Linux. This worked perfectly.

  4. Plug Drobo into Linux and format 2TB LUN to ext3.

  5. I backed up about 1TB of data and put it on the Drobo under Linux (my laptop backup), and verified that I could indeed read the data from the disk, by restoring some large VMware images back to the laptop’s disk and running them there. This worked perfectly.

  6. I shut the Drobo down cleanly, powered it up and ran a full disk check on the filesystem on the Drobo (using e2fsck from the Linux host). No issues or errors were detected on the volume.

  7. I plugged the Drobo into Windows, and through Drobo Dashboard, upgraded the firwmare to the latest available version (1.3.5). Since Windows could not read the filesystem itself, I cleanly shut down the Drobo after the firmware upgrade.

  8. I unplugged the Drobo from all host computers, and plugged it back in. It’s dead. :frowning:

What I see now are 3 blue LEDs from the right side, 1 blue LED from the left side, and a blinking green light on the bottom. No drive lights at all (no red, no yellow, no green).

A pretty serious bug exists in the Drobo here, because without a complete boot, the fan doesn’t actuate, to cool the drives which are spinning and doing something, but what they’re doing, I can’t tell. I had to angle my “Vornado” fan right into the front of the disk pack to cool it down; the Vornado moves a lot of air. :slight_smile:

The lights look like this:

…(-) ( )…
(o) ( ) ( ) ( ) ( ) ( ) ( ) (o) (o) (o)

  • = blinking green
    o = solid blue

So what exactly is it doing, and what do the lights mean? It’s been sitting there for about 20 hours in exactly that state.

i would suggest you are DRI to troubleshoot this for you - they cant complain because it is a supported configuration :slight_smile:

then i guess whatever solution they come up with - you can apply to the disk pack you value

My question was more generalized… what do the lights mean? Specifically:

  • What does a blinking, green “power” light at the bottom mean?

  • What does it mean when there are solid blue LEDs from both sides?

Good report.

I have two follow-up questions:

  1. Did you format using the Linux Dashboard, or through another means. (This one is mainly pure curiousity)
  2. What happens when you format the pack with NTFS or HFS using Windows, then upgrade the firmware? (This would point to a problem with your unit somehow, since NTFS and HFS are supported configurations)

To answer your questions.
As I understand it, blinking green Power light means “I’m busy doing something not related to client access.”
And during the boot process, the a blue light “drips” from the left to the right, “filling the meter” presumably to show the progression of boot.
I have noticed significant variance in the timing between the “drips” - meaning that the boot progression is definitely not linear in nature. In particular, the 7-light stage seems to be slowest, at least for me, and especially when I undergo a “dirty” powerdown (ie, pulling the power plug).

[quote=“bhiga, post:4, topic:986”]
I have two follow-up questions:

  1. Did you format using the Linux Dashboard, or through another means. (This one is mainly pure curiousity)[/quote]

I created the LUN using the Windows Drobo Dashboard, and formatted it under Linux, using drobo-utils there.

[quote=“bhiga, post:4, topic:986”]
2. What happens when you format the pack with NTFS or HFS using Windows, then upgrade the firmware? (This would point to a problem with your unit somehow, since NTFS and HFS are supported configurations)[/quote]

I haven’t tried that, but then I wouldn’t be able to use it, since NTFS and HFS+ won’t be able to hold a backup of my systems, because it lacks the file-naming semantics. I could certainly try to copy some junk data to it, and test that.

[quote=“bhiga, post:4, topic:986”]
As I understand it, blinking green Power light means “I’m busy doing something not related to client access.”[/quote]

So is it touching data? Or the “management” side of the disk pack?

[quote=“bhiga, post:4, topic:986”]
And during the boot process, the a blue light “drips” from the left to the right, “filling the meter” presumably to show the progression of boot.[/quote]

This is where I’m confused. Light 1 and lights 8, 9 and 10 are lit.

Lights 2-7 remain off. So what is it actually doing?

ok here is what i know:

1970-01-01,00:00:34: Cache init complete […**]
1970-01-01,00:00:34: disabling event handling [
…***]
1970-01-01,00:00:34: DiskManager LoadDiskPack called

the stars and dots at the end are the blue LEDs “*” = On, “.” = Off

@ hacker: bhiga was only suggesting you try it with ntfs, if that fails too then you evidently have a faulty drobo It is just a troubleshooting step, not a suggestion fir a usable configuration

@hacker Have you contacted tech support about this yet?

So the device is in “disabling event handling” stage? For 20+ hours? What else could it possibly be doing between that step and the next phase?

@Jennifer: No, I have not contacted support about this issue, and I probably will not, if my recent experience with support has been any indicator of success.

My last communication with support on my current open case was 4 days ago, and they’ve pretty much walked away from the incident without any closure.

I’ll pull the disk pack and keep running some tests before I fall back and try to contact support about this.

It is absolutely clear that even with a supported, 2TB ext3 volume on a clean disk pack, the device does not complete booting when upgraded from 1.3.1 to 1.3.5, although in this particular case, the lights displayed are different (1, 8, 9, 10 lit, but my actual data disk pack has 4-10 lit).

This confirms my assertion that the 1.3.1 -> 1.3.5 upgrade on a disk pack formatted to ext3 (of any LUN size) is a problem, and the Drobo will refuse to boot it after the upgrade.

More to come as I dig into this a bit more…

i think its more that the event handling bit has completed. it is the next bit which fails, the load disk pack

@hacker:
Don’t condemn someone for not helping without asking for help first.
This may be an entirely different issue, which is why I asked about doing an NTFS test.

It’s like buying a car, then you install an aftermarket engine part on it, and the car doesn’t work properly. The car dealer isn’t obligated to help you with the aftermarket engine part.
However, if you remove the aftermarket part and put the original part back into its original from-factory configuration, they will have to help you with it - or you’ll find out that the aftermarket part is the problem.

If your Drobo does not function properly in a supported configuration, then there’s something wrong with it that is covered by warranty and I highly doubt support would refuse to help you at that point.

I am using a supported configuration, which is exactly why I am trying this series of tests.

If I partition the device, format the device and handle the device per the supported specifications and guidelines (not “installing any third-party car parts”), and it still fails, in exactly the same scenario as my “unsupported” configuration (the only difference being a 2TB LUN vs. an 8TB LUN), then I should still fail under the support requirements.

At this point, I’m trying to do a “double-blind” study to verify that my original assertion was correct. In 3 out of 3 tests, I’m getting exactly the same results:

A Drobo with (any size LUN formatted to) ext3, running 1.3.1 will run and operate flawlessly through several independent shutdown/reboot phases, on two separate operating systems.

The very first boot after upgrading the firmware to 1.3.5, will result in an immediate failure, and the disk pack will not complete booting, nor will it present a block-level device to the host OS (Windows or Linux).

The next step in my test, is to do the following:

  1. Shred a clean set of disks (shredded outside of the Drobo to remove the volume header information from each disk in the disk pack)

  2. Install firmware 1.3.1 onto my Drobo v2 unit

  3. Insert the disk pack to cast that disk pack as a 1.3.1 disk pack

  4. Shut down and pull the disk pack from the unit

  5. Image each disk in the disk pack

  6. Upgrade the Drobo unit to firmware 1.3.5

  7. Insert the disk pack (thus upgrading the disk pack to 1.3.5)

  8. Shut the Drobo down

  9. Image each of the disks

  10. Compare the images byte-for-byte

From this point, I should be able to tell exactly what differs in a 1.3.1 disk pack vs. a 1.3.5 disk pack, and with this information, I may be able to “downgrade” the disk pack.

If I put this 1.3.5 disk pack into the 1.3.5 Drobo, I know it will fail to boot. If I can “downgrade” the disk pack to 1.3.1 and successfully boot the disk pack, it will be further confirmation that the 1.3.1 -> 1.3.5 upgrade path is what is causing non-function from an ext3-formatted disk pack.

Make sense?

Your downgrade testing sounds good.

What my thought is, though, is that there’s a possibility that there is an issue between your Drobo unit and firmware 1.3.5, independent of the disk pack’s volume, so I wanted to establish that your Drobo with firmware 1.3.5 can load a disk pack successfully.

In your orignal steps 4-7, if you format as NTFS first, then do the data integrity test, after step 7 (upgrade firmware to 1.3.5) and a Drobo reboot, if you can establish whether the NTFS disk pack does or does not mount and read/write properly in Windows, then you’ll know with certainty whether the issue is with your disk pack’s format or not.

Getting out of Linux realm keeps you out of the gray “beta” area.

[quote=“bhiga, post:13, topic:986”]
Your downgrade testing sounds good.

What my thought is, though, is that there’s a possibility that there is an issue between your Drobo unit and firmware 1.3.5, independent of the disk pack’s volume, so I wanted to establish that your Drobo with firmware 1.3.5 can load a disk pack successfully.[/quote]

I have tested that a 1.3.5 Drobo having seen no disk yet, can have a disk pack created within it, formatted to NTFS, and reboot fine numerous times. I did not test a 1.3.1+NTFS => 1.3.5+NTFS round for completeness, however.

Why not… I’ve got a large pile of 1TB disks here now for exactly this kind of testing. All it means is spending more time digging into it, while my original disk pack sits idle until DRI can produce a fix that allows it to boot.

Beta, grrr… It’s not like ext2 or ext3 is a “new” technology. It’s only been around for over 16 years.[hr]

You know, I think you might be onto something here…

I tried something that was suggested to a Windows Drobo user in another forum, and managed to get my Drobo to give me 4 solid red lights, 10 solid blue LEDs, and 1 solid green light, with my original non-working disk pack.

My Linux host OS sees the physical device (/dev/sdc), but not the partition it contains (/dev/sdc1)

So what does this set of lights mean? I don’t see anything in the manual about this at all.

Blinking drive lights, sure. Solid greens and reds, sure. But ALL of the lights at once? Nothing.

I don’t know, I only quoted from my log file :wink:

10 solid blue?
Sounds like maybe it got to a different point in the boot process and is confused about the free space calculation and showing completely full. Being that Linux sees the physical device, it sounds a bit hopeful.

By chance, in that state, have you tried a Windows client with the ext2 driver? I have not tried the ext2 driver at all myself, so I have no clue to its safety or overall compatibility. Definitely do it on a copy of the pack and not the original.

[quote=“bhiga, post:16, topic:986”]
By chance, in that state, have you tried a Windows client with the ext2 driver? I have not tried the ext2 driver at all myself, so I have no clue to its safety or overall compatibility. Definitely do it on a copy of the pack and not the original.[/quote]

I didn’t try that, but since it still isn’t handing the block-device’s partition over to the host OS, Windows wouldn’t be able to do anything with it either… until /dev/sdc1 is seen by Linux, or that same GPT partition is seen by Windows, the native ext2 driver on Linux or the ext2 driver on Windows, won’t do any good.

I’d be curious to know if your drobo can upgrade an NTFS disk-pack from 1.3.1 to 1.3.5 and then boot with it ( i suspect it can since i would think this configuration gets a LOT more testing)

However from your failure to upgrade a support 2Tb ext3 from 1.3.1 to 1.3.5 it sounds like maybe they didnt test 1.3.5 with ext3 at all. i would definitely get back onto support - this is a support configuration, and it is failing. so maybe if they can fix the supported configuration then your old disk pack will start to work too??

Im thinking more is 1.3.5 for 2tb ext3 is borked - if they fix this in 1.3.6 so it can boot 2tb ext3 (as they should do!) then you may find that it will also boot your 8tb ext3?