Firmware 1.3.6 Seems to have mucked up things

Forum History Drobo
1

I got the automatic update pop-up for the Firmware, so I decided to update it (not the first time I have done that, and in the past it seemed to work fine).

So when my unit booted back up, now one of the drives is marked as bad. Also The unit seems to show 100% usage, which it says is 3.6TB of 3.6TB on the dashboard, but before I installed the Firmware, there was less than 2TB actually used. I believe that something is rotten in the state of Denmark.

So I am wondering…
[list]
[]Has anyone else encountered this problem?
[]Is there a good way to rollback the frimware, and how can I get a copy of that firmware patch?
[]Should I think there is any chance the drobo will find the old data if I rollback the firmware?
[]Should I replace the “bad” drive?
[]How long does Tech Support take to answer something posed on the support site usually?
[]Am I just screwed? :slight_smile:
[/list]

If it matters, this is my setup:
Drobo second gen with DroboShare
4 drives, all Western Digital Caviar Greens. 2 of them are 2TB and 2 of them are 1TB. the one now marked failed is 1TB.
--------
Edit: I thought because this seems to be a known problem, I will post more info here and update with my eventual Fix.
--------
Here is my setup:
Drobo 2nd Gen
Drobo Share
The Drives are:
[list=1]
[] Western Digital WD20EADS - 2.0TB (caviar green)
[] Western Digital WD20EADS - 2.0TB (caviar green)
[] Western Digital WD1001FALS - 1.0TB (caviar black)
[] Western Digital WD1001FALS - 1.0TB (caviar black)
[/list]

I connect exclusively via droboshare.

Before the firmware patch, my drobo lights were normal four green drive lights and 3 blue lights along the bottom, and DroboDashboard showed correctly 1.25TB usage for 35% of the drive.

After the Firmware patch, one of the drive lights turned solid red and all the blue lights were on, and DroboDashboard showed 0 bytes free and 3.600TB used. The drive and data were inaccessible.

Working with Tier 1 tech support, I connected my drobo via USB dirrectly to my Win7 box. Booting up there, the Lights returned to normal as before the patch. I was able to access the data. Tech Support got me to run chkdsk (notes on chkdsk below) several times on the drive until no errors occurred and try again on the DroboShare. I got the same bad results on the DroboShare. So the end is, my Drobo now works without the DroboShare and I am being escalated to Tier 3 Tech Support.

So when my unit booted back up, now one of the drives is marked as bad. Also The unit seems to show 100% usage, which it says is 3.6TB of 3.6TB on the dashboard, but before I installed the Firmware, there was less than 2TB actually used. I believe that something is rotten in the state of Denmark.

For a little more detail on the lights as my drobo boots up on the DroboShare:
[list=1]
[] All solid red on right, all solid blue on bottom (for a few seconds)
[] All lights out (5-10 sec)
[] Blue lights scroll from left to right stacking. Some pauses as that completes, but completed in maybe 10-15 sec.
[] Blue lights out, 4 greens on right hand side
[] 3 blues add at the bottom from left to right until all 3 expected are lit. Note: This is how we expect the lights to be. This lasts for a few seconds.
[] Then All Blues light up and third Dive light turns solid red.
[/list]

Chkdsk:
Tech Support wanted me to run “chkdsk f: /x”, where “f:” is the USB mounted drive letter.
chkdsk apparently may have problems in Win7 and Vista on 16TB drives, which is how I configured my Drobo, so it will run correctly for 10-15 minutes then spin off into la-la-land, where the CPU on your computer will fluctuate, but no disk I/O happens and stay in that state forever. It would actually get to the last line when it should be over for me and jsut sit there. I had to kill the chkdsk process. However, running it through several times did eventually correct all the file structure problems on the disk, but this apparently had nothing to do with the original problem.

[size=x-large]Solution:[/size]
After a bit of jerking around, I was given a firmware patch for the DroboShare (v52SledUpgrade.tds). This cleared up the problem, but not without a little strife. After I applied the firmware, the softboot following did not make the Drobo come back up. The Amber “Standby” light was the only light on (under the faceplate). So I just cycled the power on the Drobo and DrobShare and that finally go me back up and working.

2

you cant rollback - once your pack has booted with a firmware its flagged as such - the only way to go is up im afraid.

it would appear a lot more full (in percentage terms) after a drive failure as you would have much lower capacity available.

how much data do you actually have on their (i.e. if you ask your operating system the total size of all files).

run chkdsk on the volumes to see if that is what is hogging the space (i.e. free space incorrectly marked as used)

3

With 2 x 2TB drives and 2 x 1TB drives you had 3.6TB available for data.
When that 1TB drive failed, you are down to 2.7 TB available for data.

You either need to look at the used space on the actual volume or go to advanced controls > DATA tab to see how much you have USED.

If you were at 2.7TB used before the drive failure, yes once that drive failed you would be at 100% full.

The firmware did not cause the drive failure, it’s purely coincidental.

If you created a case online, as soon as we open we start grabbing cases in case number order and working them. We open 6am Pacific Time.

You need to replace that failed drive.

4

Ok, I did open a case. See, the thing is, there was less than 2TB of data on the drobo, so something else is wrong, but I can’t connect to the drobo filesystem to scan and see how much is there.

So, I’m looking back at my records, there was probably 1.7TB of data on the drobo drive, but now drobo dashboard is saying 3.60TB is used.

I will acknowledge that the drive may have coincidentally gone bad, but the drobo seems to be reporting wrong sizing info anyway, so I dont’ suspect it is a coincidence. I’m unsure if I should really replace the drive yet, because I’m afraid the drobo might not be seeing the data on the other drives correctly right now, so I’m not sure what will be lost.

5

I just upgraded and have just had a scary few minutes where it was reporting way less used space than it had before the update.

As I started writing this it’s updated itself, and is now reporting the correct value.

Phew! Thought I’d lost a lot of data there…

6

Then I would recommend directly connecting your Drobo to your computer and running CHKDSK on your volume.

When was the last time you ran CHKDSK on your drobo?

7

Quick question, when you say it was flagged as bad, was it blinking red or is the drive solid red?

8

I didn’t even know you could run chkdsk on a drobo. I will try this and report back.

The light is solid red.[hr]
ok, so, connecting directly to the PC’s USB port, the drobo comes up clean and all drives are green. the space used is 1.25TB and space free is 2.35TB

9

Solid red light is not a failed drive.

10

So, The support Ticket guys told me to run chkdsk, so I have been, but before I did the chkdsk, I decided to just browse files (connected locally on USB). Everything looks fine connected directly USB, but chkdsk seems to be stuck for many hours on the same point. it got to this point in about 5 minutes and then stayed:

C:\Windows\system32>chkdsk f: /x
The type of the file system is NTFS.
Volume dismounted. All opened handles to this volume are now invalid.
Volume label is Drobo.

CHKDSK is verifying files (stage 1 of 3)…
205184 file records processed.
File verification completed.
143 large file records processed.
0 bad file records processed.
0 EA records processed.
0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)…
253306 index entries processed.
Index verification completed.
CHKDSK is scanning unindexed files for reconnect to their original directory.
4 unindexed files scanned.
CHKDSK is recovering remaining unindexed files.

I’ll see in the morning, if it’s still stuck, I’ll kill the chkdsk and try it again on the droboshare.

11

chkdsk should be run on directly connected drobos. (i dont think it will even work on network shares)

You can always run chkdsk on drobo. Drobo only redundantly protects you against hardware failures. It is perfectly possible for drobo to be redundantly protecting your corrupted file system (if you see what i mean).

12

chkdsk still not complete with no discernable progress since 5 minutes after it started.

The Drobo seems to work with no disk errors showing and 3 blue lights (none blinking) when it is hooked up to the USB port directly, but when the drobo is booted up connected to the droboshare, the drobo comes up in an error state with one drive light in solid red and all of the blue lights on.

13

i’d add that to your support ticket and send them the logs and see what they say

i have never had a droboshare - does it share the drobo’s power supply? it almost sounds as if drobo doesn’t have enough power to power up the last drive properly/fully when its connected to droboshare?

14

Sometimes Windows 7 has issues chkdsk’ing 16 TB volume sizes.

Can you try on a Vista/Server machine?

15

They do share the same power source, but it has been working fine for a little under a year until yesterday when I applied the firmware update to it.

16

I have one vista box still, and chkdsk seems to be doing something similar, though it did seem to get a little farther (not sure if that’s just because it already did some before), but only just a little farther.

Here’s the current run in case it spawns a thought for someone. Once again, it got as far as it did in about 6-7 minutes, then it has remained in this state with no drive sounds for about 4 hours.

[code]C:\Windows\System32>chkdsk e: /x
The type of the file system is NTFS.
Volume label is Drobo.

CHKDSK is verifying files (stage 1 of 3)…
205184 file records processed.
File verification completed.
143 large file records processed.
0 bad file records processed.
0 EA records processed.
0 reparse records processed.
CHKDSK is verifying indexes (stage 2 of 3)…
Deleting index entry found.000 in index $I30 of file 5.ocessed)
253308 index entries processed.
Index verification completed.
CHKDSK is recovering lost files.
9 unindexed files processed.[/code]

Does anyone know how long a chkdsk like this should take?

17

One of my customers in Japan market reported similar today.
Drobo Gen2 connected via DroboShare with 1TB + 1TB + 1TB + 160GB、populated with about 20% data. After the new firmware update, it now reports disk full and data is inaccessible.
Replacing 160GB drive with 1.5TB drive increased the total volume but still reports disk full.

18

that would be a case of sending support your logs i suspect.

19

Yes please open a support case.

20

I updated my original post with more specific detail that might help someone else who has this issue. I will also update here as I go through Tier 3 Tech Support and hopefully solve the issue.