I’m in a scary ‘all red lights’ mode on my Drobo FS. Note that I’ve searched and can see others with red light issues, but no one with the same issue as this.
How I got here (this happened over several days):
My Drobo was getting full and had 4 x 2TB drives in it. It was acting slow so I decided to upgrade the size. I have dual disk redundancy on.
I added a single 4TB blank in the last slot. The Drobo went to data protection mode, flashed for a long while (must have been over 12 hours) and then eventually came back.
Once it was all green again, I took OUT one of the 2TB drives and replaced it with another 4TB blank. So now I’m sitting at 3x2TB and 2x4TB. Again, the thing went through data protection and eventually came back and mounted.
This is where the weirdness started. The Drobo shares stopped mounting consistently, and I had to run through a lot of reboot cycles. I noted that it seemed the Drobo would get to a point, be accessible, then would drop into a weird mode where you could see the Shares but they all looked empty. Rebooting would bring it back and I could see the data again for a little until it cycled into a weird state again. Different drives would show as red (with all others green) on each reboot.
Eventually it just locked into a data protection mode and and sat there. Shares could be seen but not with any data. The size estimates seemed right though.
This is where I made the probably bad decision, and accepted the Dashboard asking me to upgrade firmware. I figured this might help. But as soon as the firmware finished updating it rebooted in the current state…
Current state: Drobo turns on, shows all yellow (maybe yellow/green) lights, then the blue lights start flashing from left to right, then all lights go red. Drobo will eventually show up in the Dashboard and says that too many drives have been removed. Capacity / Used / Free all show as 0 B. Firmware is now showing as 1.2.7 [4.45.10288]. Drobo Dashboard is 2.7.0 [77097].
What I think happened: I think something weird was corrupted in the firmware meaning the shares weren’t mounting properly, and when I updated it just messed it up further. I think my data is fine but have absolutely no idea how to go about recovering it. I have read things about removing drives and stuff, but don’t want to rush into that without understanding what that triggers.
Questions? Suggestions? Need more info? If anyone can help me I would be infinitely grateful.
hi, i was making some notes as i read through so apologies if i misunderstood anything
from what i can see, is that you had 4x2tb drives -2tb -2tb (for ddr) and essentially had about 3.6tb usable space. (4tb *0.9 filesystem overhead)
and then added a 4tb drive = +4tb = 2+2+2+2+4 (-4tb -2tb ddr) = 6tb (*0.9) = approx 5.4tb usable
then took out a 2tb drive and replace with 4tb = 2+2+2+4+4 (-4tb -4tb ddr) = 6tb raw (*0.9) = approx same 5.4tb usable
(updating firmware can seem unfortunately slightly risky in itself, but probably more so if the drobo is not in a stable state at the time) but can i check that when the last rebuild took place, were things ok for a little while, but ‘then’ shenanigans happend… or if shenanigans happened immediately after the rebuild?
(there may be a way to re-apply a firmware, in case it was not applied fully, but you are right to slow down and check things before rushing in so no worries there… and if you have a but more info on what the mounting problems were, it may be handy to anyone else on the thread too, such as if you had to modify any dhcp settings or share renamings etc)
Thanks for the reply Paul. You seem to have got it right. Unfortunately I don’t really know what the rhyme or reason was with the issues, my memory is foggy on that part of the issue. Because the whole Drobo was so slow before I did the drive swaps I was distracted to other issues, but I think it wasn’t even mounting consistently back at that time. That is, the shenanigans with mounting were happening for a while.
In more detail, I think the Drobo would sometimes mount shares, and sometimes not, and then sometimes once the shares were mounted I could access them for a while, but then later I could come back and they would show as empty. Unmount and remount would bring them back, sometimes. I pegged it at the time to the ‘drobo nearly being full’ issue and figured the addition of drives would help.
I know my memory isn’t perfect on this, but I can’t change that.
I guess what I’m trying to figure out now is in the current state, what would be my next step? Run some sort of diagnostics? Do some further digging to try to figure out the issue? Remove drives and put them back in? Try to re-flash the firmware? I don’t know how risky any of this is and if I could cause further issues needlessly, or worse, trigger a reaction in the box that wipes the data. I want to obviously avoid a step that will risk the data.
I found a special Drobo recovery place that quoted me $2,000 to $4,000 for recovery, which I cannot really afford, so hoping to hack through this myself (with help here) as much as possible.
thanks for more info,
ok thats ok about the history of usage, it sounds like it could have been one or more drives going bad, as i also experienced some sluggishness and shenanigans on my drive failure thread here: http://www.drobospace.com/forums/showthread.php?tid=144519
ideally, if dashboard let you generate diagnostics logs, sending those to support would be good to try and determine which drives may be the ones causing the all red state (eg usually you need a certain number of error-free drives to be able to access the data spread across them, but some maybe more known to be perfect yet still need the others) though am not sure if any extended support/pay per ticket is available for the fs - you could try to see though but am not sure…
so the next best thing could be to try the following:
to power all off (drobo and computer)
to remove all drives from drobo (remembering the order)
to unplug the connection cable
to power up the empty drobo on its own
(does it go through the usual led boot up sequence, and then goes into standby mode?)
if so:
to then boot up the computer
and once up and running, to plug in the drobo connection cable
the drobo should then wake up
(what does dashboard show now? does it find it, and prompting you with a top bay light red there, as well on the physical device?
If so, its probably meaning that the hardware is ok, which is good in itself, but probably means that the drives have errors…but, if you then power all off, and then plug in the drives back to where they were still with power all off, and then boot up the drobo on its own without connection cable, what does it do now devon?
Thanks again Paul, I got the diagnostic file downloaded and sent off a ticket to Drobo. Hopefully they will respond.
I was considering the risk of removing all drives, and something just struck me. The Drobo Dashboard is saying ‘Too many hard drives have been removed. Please re-insert the removed hard drives.’ I ignored this at first because I thought it was the issue with installing the firmware. But now it’s struck me that only one drive was removed during the whole process, and maybe I should simply re-insert it. To review, here are the states the Drobo has gone through:
State A - Initial State:
[Bay 5] Empty
[Bay 4] 2TB #4
[Bay 3] 2TB #3
[Bay 2] 2TB #2
[Bay 1] 2TB #1
*two new empty drives, 4TB #1 and 4TB #2 are outside Drobo at this state
State B - Added 4TB #1
[Bay 5] 4TB #1
[Bay 4] 2TB #4
[Bay 3] 2TB #3
[Bay 2] 2TB #2
[Bay 1] 2TB #1
*one new empty drive, 4TB #2 is outside Drobo at this state
State C - Removed 2TB #4 and replaced with 4TB #2 (all in one quick action)
[Bay 5] 4TB #1
[Bay 4] 4TB #2
[Bay 3] 2TB #3
[Bay 2] 2TB #2
[Bay 1] 2TB #1
*freshly removed 2TB #4 is outside Drobo at this state
Proposed State - Remove 4TB #2, re-insert 2TB #4, returning Drobo to State B
[Bay 5] 4TB #1
[Bay 4] 2TB #4
[Bay 3] 2TB #3
[Bay 2] 2TB #2
[Bay 1] 2TB #1
*4TB #2 would be outside Drobo at this state
Does this plan make any sense at all? If so, there are questions:
Is waiting for Drobo support to check my diagnostics worth it?
Better to do it with power on or power off?
Most importantly, should I do it all at once (remove 4TB #2 and re-insert 2TB #4) or wait between these actions (and maybe power on to allow it to figure itself out).
I think #1 is no because there was only one drive removed that could be the problem, #2 is power off and #3 is all at once. But that is really a guess, so maybe someone could give input on actual knowledge or experience with the Drobo’s internal process? I think the exact movements are important here.
ok i still think its good to ascertain if the hardware (or drives) are the problem, which would be via the last post i made about powering down, removing drives and powering up empty, (and as long as you Only put them back in after the test while power is OFF, it should be ok)
Then if the problem is with the disk pack, then you could try the disk pack drive cloning process via yerry’s thread here: http://www.drobospace.com/forums/showthread.php?tid=144867
(im hoping that it should be considerably cheaper than the cost you described)
if you would prefer to not try that, then i think if you had to choose only one of those options, then it might be probaly better to try powering off first, and then putting that drive that was there before, back in, and then powering up. (what does drobo and dashboard do?)
if it still didnt do anything, (you could try another reboot without any computer cables too just in case)
whenever i have replaced a drive (or upgraded a drive, i usually try and wait for drobo or dashboard to recognise that a drive has been removed, and then i put the new one in, unless it started rebuilding then to wait until stabalised)
Paul, I’ve gone ahead and tried your emptying the drives suggestion. With an empty disconnected Drobo, this is what happened when I powered on:
all lights went on
blue lights turned off
blue started going across bottom, but didn’t finish
single red light in the top bay
What does this mean to have a single red light in the top bay?
Drobo Dashboard just says ‘Drobo does not detect any hard drives. Please insert a hard drive immediately.’ Firmware is still showing as 1.2.7 [4.45.10288].
Any thoughts? Maybe I could upgrade the firmware? In the meantime I’m going to re-insert all the drives in their final state (State C above).
thanks for trying that,
usualy an empty drobo will start to bootup, and then go into standby mode. (the top bay red light is ok and usually the 1st slot that people put a drive into, and signifies that a drive needs to be inserted)
if you now connect it to the computer, dashboard should see it and recognise the empty drobo.
(does it do that?)
Yeah it did that, I should have mentioned in my previous post. The Drobo Dashboard shows the same thing, says the Drobo is empty, and asks to insert a drive. I’ve since turned it back off, filled the bays back up and get the same response as before.
ok thanks devon, it seems like the fs is itself ok, and it might be some problems with the disk pack.
when a drobo has been booting up/rebuilding and then rebooting in a loop, lots of recommendations have been to fully clone the drives via that process above. in your case though, while it is not actually looping, one thing you could try if you can, would be to try cloning that drive which was initially inside, before putting it back in (just in case it was to get wiped)
eg to clone it before i mentioned this:
‘if you would prefer to not try that, then i think if you had to choose only one of those options, then it might be probaly better to try powering off first, and then putting that drive that was there before, back in, and then powering up. (what does drobo and dashboard do?) if it still didnt do anything, (you could try another reboot without any computer cables too just in case)’
as far as i remember, powering off and putting a drive in, and then powering on again, didnt usually wipe the inserted drive (as opposed to a normal hot plug process), except for certain combos of firmware when done just after resetting some drobos, but as am not 100% sure then maybe making a clone if possible would be safer to try with first.
Thanks for the info Paul. Is there anything else to try without cloning? I can’t really afford to go buy a full extra Drobo and drives…
I tried to get Drobo to decrypt my diagnostics, because all I really want is to know which bay it is that’s returning errors, but they just gave me a form response asking me to pay for service. Quite ridiculous.
hi devon,
thats ok, while its good to have some backups or replication if possible, it does add up the costs and/or reduces overall usable capacity too.
(i’m actually glad i had my local gen1 backed up to my gen2 too, as the gen 1 started playing up but thats another story for another thread)
what i was actually meaning when mentioning cloning, was just the bare drives, instead if needing to buy drives plus main unit if that makes it more feasible?
I actually think I have a pretty good solution. I had thought my replicated backup (I have one on site and one off site on external drives) were also corrupt, but I was finally able to restore with help from their customer service (am using Crashplan). So the short story is I now have all my photos recovered (which was all that was redundantly backed up), which means I still lose all my media (music, etc.) and a bunch of random stuff I had collected over the years, but that is not even close to the same level of importance as my photos (hence why they weren’t redundantly backed up).
So I’m thinking at this point, it may be time to pull the bandaid, format my Drobo, and copy all the photos I’ve restored back onto it, and call it a day. I’m assuming that’s just through a handy reset in the dashboard, but any cautions?
hi devon, its good to hear that you have recovered your photos, and it may well be that resetting/reformating addresses things, but i guess the main cautions would be that formatting essentially wipes the drives, and that it might be something else (such as drives / diskpack) which have more problems down the line. the only other option which could be available would be to try raising a legacy support ticket (paid incident i thinks about £60 or thereabouts) where they might be able to ascertain which drive(s) are bad, and you could possibly just get that one cloned.
Thanks for all your help Paul. It’s unfortunate, but I cannot accept the fact they refuse to simply tell me which drive is faulty. I find this unacceptable coming from a hardware / software provider in 2016, so in the short-term I will reformat my Drobo as a file archive, but I will be making moves to leave Drobo in the dust once and for all. I will be writing a Medium post, sending it to my full network in the tech community, and then make a post here referencing it. Sad to see such poor support in this age.
Wish me luck in the research of a better supported NAS platform!! And thanks again for the support from you in this thread, it was helpful to exhaust all the options. Just sorry to see the product isn’t supported with as much pride from the company side.
hi devon, thats fair enough
i think upgrading to a 5n gives you some migration support which could help, and being newer also has more features for apps etc (though you would need spend something eg to buy a 5n)
if you still have the unit and drives as is (eg and have not reformated yet), it may be worth still trying a manual re-application of firmware (either the same one that you installed last, or to the latest one shown on the website here, just in case?) and then to also try a reformat test while empty. http://www.drobo.com/drobo-downloads-2/
for example 1)
if you power down, remove all drives, and then try to apply the firmware again,
and then to reboot once (still while empty) to see that dashboard can still recognise it.
+
and then to power all down and to put the drives back in
and to power up again to see.
and for example 2)
if you are going to reformat those drives now anyway, then something else you could try would be this:
if you power down, remove the drives, and aply a full pinhole reset of the drobo
and then to let it boot up still empty to see that dashboard finds it (see note ALT below)
and then to apply the firmware update, and to reboot to verify the version with dashboard
and then to power down again and to put the drives in and to then power up to test
(there is an extra possible step whereby you could alternatively (ALT) power down and put the drives in Before the firmware update, just in case the base factory version helps in some way, but i think some firmware is saved as part of the disk pack but could be worth a final test)
whichever way forward you choose, if you do have any luck with the other ideas above please let us know (or if you just fancy some general computer chats etc on other topics then feel free to drop by sometime)