[Solved] Drobo FS: File System Maintenance In Progress... for a long time...

Drobo - End of Sale Drobo FS
1

So, my local power company decided to swap out my meter on thursday. I didn’t powerdown my drobo FS.

For the record, it has 5 2tb drives in it, and when I last checked it was showing 11% space available. yeah, the top bay is yellow-lit.

So, after power came back up, it failed to mount. It’s showing up in the dashboard (on three computers) just fine, but it was showing the “Internal Problem” message in the dashboard. I tried to follow instructions here (http://myproducts.drobo.com/article/AA-01333) to start file system repair. It’s still going.

The drobo is responding to the dashboard, I can flash lights, etc. but Shutdown, Restart and Reset are all greyed out in the drobo dashboard > Tools, and the Capacity screen and the Shares screen both show “File System Maintenance In Progress” message. It’s been that way for two days now.

What has me worried is the fact that for most of the day thursday and friday morning, the drive’s activity light was going and I could hear the drive heads moving. Now, silence… and the activity light is off. about once every minute or so, i’m seeing a quick flash from the green activity LED, and then nothing. No drive noise.

I’ve been re-reading the post here: (http://www.drobospace.com/forums/showthread.php?tid=141627&page=1) and I’m nervous if I have the same problem SJSteve and Bradii had to resolve… I’m very good at learning stuff quickly, and I actually have a second empty Drobo FS chassis AND spare test drives to play with… but I don’t yet know where to start. How do I get to the dmsg Log? how do I start to communicate with my drobo to get going on this? Should I wait longer?

If it’s a total loss, I would be very sad… 5 out of the 6tb on the drive are backed up to single drives, but that last tb would be annoying to have to rebuild/collect.

Can anyone give me some direction?

2

So, I’ve found this: https://sites.google.com/a/droboports.com/www/app-repository/nbd-3-7/drobofs … Is this a product the thread above? (http://www.drobospace.com/forums/showthread.php?tid=141627&pid=180138#pid180138)?

I still don’t know if I need to go that far… I haven’t yet talked to my drobo… I need to SSH into it first and see if it’s complaining about multply-claimed blocks… If not, I should probably just leave it alone to finish the repair on it’s own…

Still trying to resolve.

3

Ok, so I’m now working on my spare DroboFS box, which I’ve named TestDrobo. I put in two spare 80g HD’s to get it up and running… (oddly, it isn’t giving me 80g available… first, it gave me 10g, then I reset and it gave me 40g, then I added a 250g on top of the 2 80’s and now it’s giving me 112g free… WTF??)

Anyway, I set it up, activated Droboapps, and dropped in Dropbear… and I’m now SSH’d into the TestDrobo. I can navigate around the file system, but I can’t seem to get any commands to work except LS and CAT… Do I need to check permissions or something? I’ll get there…

4
  1. When you type “whoami”, what do you get back?
  2. When you type “echo $PATH”, what do you get back?
  3. When you type “uname -a”, what do you get back?
  4. What is the firmware version of the Drobos?
5

Ricardo, Glad to see you responded. Before I answer your questions, let me give you an update on the main drobo FS…

I got tired of waiting, and based on Bradii’s comment that he power cycled his FS several times while working out a solution, I went ahead and powered down my FS using the switch on the back. I popped out the drives. I powered it back up. I let the dashboard install the latest firmware (1.2.7? i think?) It was on 1.2.6. I let it restart and verified the firmware number changed to 1.2.7. I powered it back down. I put the drives back in. I powered it up and waited.

It eventually mounted the volume! However, the dashboard scares me because it’s reporting I am using only ~300 megs out of 7.21 terabytes. That’s definately not right… however, I was able to connect to the shared via my desktop and I was able to play a random video file… so, I think all the files are there. I suspect I need to go ahead and immediately copy the 1tb of data that ISN’T backed up onto additional HD’s (I just ordered 2 4tb drives for this, this morning… and a UPS).

Am I in danger if I let anything write to the volume? should I go into shares and lock everything down to read only? would that protect me enough? I’m very nervous right now since things aren’t perfect… I don’t want to run repair again, since it’s a dangerous action…

I went ahead and powered down the drobo via dashboard’s powerdown command. it remains off until my backup drives are ready to receive data off the main drobo.

So, based on where I am now, any thoughts?

6

Ok, another update. The information above was somewhat premature.

So, I powered the FS back up today, and I saw the “Used” space on the Status screen start out low and I realized it was slowly climbing back up. after letting it sit, it’s now showing 6.39 TB (89%) exactly as I would expect… (out of 7.21TB)

…however, now it failed to mount again. This time instead of the “Internal Problem” message, I get the “Mount Failed” message on the Capacity and Shares screen, with the text “Drobo detected an internal problem. Please contact Drobo Support by clicking on the link below:”…

So, after letting it start all the way up and letting it stabilize, I restarted it. and it came up the same. I guess I’m going to run Drobo Repair on the Tools page again, and cross my fingers. I’m hopeful since I actually DID see my files this morning… I should have left it on and just started copying files. Sigh.[hr]

ok, answering these questions… .this is on my testdrobo box.

whoami

root

echo $PATH

/usr/bin:/bin

uname -a

Linux TestDrobo 2.6.22.18 #1 Wed May 22 16:43:02 PDT 2013 armv5tejl GNU/Linux

firmware on the testdrobo is still 1.2.6

7

So, I’m trying hard to figure this out on my own; but I could really use some direction…

My FS is still showing Mount Failed, but at least it is registering the correct amount of used space. I’ve run Drobo Repair twice, and Drobo’s support guys says to NOT run it more than twice.

I’ve tried the method of reinstalling the firmware, and that sort of worked once, but i powered the drobo down and restarted it, and the mount fail problem came back.

I REALLY want to try the method used by SJSteve (http://www.drobospace.com/forums/showthread.php?tid=141627&pid=179383#pid179383 )to find out if I have multiply-claimed blocks, but I don’t know how. yet. I would really love some help to figure out how to get into my troubled FS box and find out.

I’ve loaded the NBD module (https://sites.google.com/a/droboports.com/www/app-repository/nbd-3-7/drobofs) on my testdrobo box to see if I can make it work, and I’m slowly piecing it together before I try it on my actual data box… but a nudge in the right direction would be nice.

  1. how do I SSH into the box if it won’t mount it’s shares? (do I pull drives, put in a temp drive, load dropbear, then put the other drives back in?? I know there’s a trick to make that work… sort of…)

  2. Once I SSH into the production box, how to I actually check for multiply-claimed blocks? I’ve tried running FSCK on my testbox, but I can’t seem to get it to actually run. I’m clearly doing something wrong.

I’ve been working on building a linux VM for use to actually do the repair, IF i can get that far. It’s been down for a week, and my wife and kids are really grumbling over the missing Plex server files. ugh!

8

hi trimble,

(for the family part, maybe a huge box of chocolates can keep them all happy, especially when the wife can tell them that too many chocolates are bad for them and that she’ll have to eat most of them to play safe) :slight_smile: - most of the chocolates that is :smiley:

jokes aside though - am not too sure about the ssh parts, but maybe the following might be worth looking at:

  • to power all off and to maybe use some of the new drives if possible, to carry out a full clone of the drives
    (just in case the repairs or multiclaimed blocks process does any harm - so that you can always re-clone and try again on the clones?)

  • or if thats not possible/feasible yet, to see if its ok to enable read-only mode on the drobo, while you try to mount it.

9

So, here’s an update.

I shut down my drobo FS and pulled all 5 drives and set them aside.

Using my spare drobo FS (i bought a second one on ebay because the price was too good), I prepared another disk pack with dropbear, and I was able to load that in my main drobo, and read the repair.log. I am now able to confirm that I have the dreaded multiply-claimed blocks problem. I have ~1000 inodes affected. I’ve been able to identify all inodes that are multiply sharing more than three blocks, and there is only 16 of them. (note for anyone else trying to prove if they have this problem: Doing this doesn’t require any really tricky work, since the repair.log file is stored with the firmware, not on the disk pack. So it’s still there after you take out the bad disk pack, and insert a temporary drive to get droboapps working.)

I’ve been trying to figure out how to set up NBD to run on the drobo such that it will still be there after I swap in the main disk-pack with the problem. I haven’t been able to figure that part out yet. I found the NBD-SERVER droboapp, but it seems to be designed to work from a working disk pack, and I need to somehow install it below the disk pack so that it will run with an unmountable file system.

my FS has been offline for over a week now, but I’m determined to figure this out :slight_smile:

10

Update. I got it. nbd-server is running on the FS, and the dirty disk pack is in.

now running fsck from my desktop using a Ubuntu livecd…

wish me luck.

11

good luck :slight_smile:

12

Update…

…and it’s done. 24 hours later, over 1000 inodes repaired. carefully disconnected the nbd-client and nbd-server, and restarted…

and it’s back. File. System. Mounted. Now i’ll be spending the next several days recheck files that should be corrupt, and backing up the remaining 1tb of videos that aren’t already backed up.

Wife already falling asleep listening to her favorite video to fall asleep to… (stand up comedy???)

I want to thank Bradii and sjsteve for paving the way, and especially ricardo for all the work he did back in late 2013 and early 2014, to help those guys, and most especially for the pre-built NBD tools posted over at Droboports (https://sites.google.com/a/droboports.com/www/app-repository/nbd-3-7/drobofs)

When I get time, I’ll try to post a play-by-play of what I went through to get this fixed, to try to fill in the gaps that I got stuck on.

13

well done for getting there, :slight_smile:

when you get time it sure would be useful to know more details about what you had to do that could not be done natively, as it could help others too.

btw i think ricardos link changed a bit when the 5n came out, and might be this one:
https://sites.google.com/a/droboports.com/www/app-repository/nbd-3-7[hr]
and here:
https://sites.google.com/a/droboports.com/www/app-repository/nbd-3-7/drobofs