Drobo

What makes the Drobo "disk pack" unique?

With the recent firmware update of my Drobo v2 from 1.3.1 to 1.3.5 locking me out of accessing my data, I have to wonder… what makes the Drobo disk pack unique?

Can I plug these drives into a 4-port SATA card and access the block-level device as a normal LUN?

Since I can’t downgrade the disk pack version from 1.3.5 to 1.3.1, I’m left with few options:

  1. Say goodbye to 1.2TB of my important data, while Drobo refuses to hand the device to the OS.

  2. Destroy the disk pack version header and hope that the Drobo re-initializes the disk pack without destroying the data on the volumes itself.

  3. Wait for a newer firmware revision that fixes the showstopper bugs in 1.3.5 prohibiting the Drobo from booting my (previously 1.3.1) disk pack.

  4. Plug the 4 drives into a 4-port SATA card and mount the volume as a standard volume under Linux, and recover my data.

  1. is something you have said very strongly that you dont want to do

  2. i think this would quite possibly lead us immediately back to 1) i think the re-init will involve re-initialising the mapping table that drobo keeps to tell it what blocks are where (i.e. it would mark the whole volume as free).

  3. is probably the most prctical solution - no doubt hurried along by you leaning heavily on them and giving them baad publicity. i would image they must support > 2Tb at some point, and hopefully when they do - your data will be accessible agani

  4. without knowing how the data is laid out (which is their proprietary algorithms) i dont think you stand a chance.

basically, i think if data recovery experts cant do it (Number 4) then the chances are you wont be able to either.

However it may be worth while giving several of them (expert recovery people) a call - explaining the issue and seeing if they can do anything about it.

even if they are unsure, since your disks are all working perfectly - they can clone your original disk pack and “play” with their own copy of your array and see if they can get sense out of it. you never know your luck, and if your data is that valuable, its probably worth a try.

I thought of another one:

  1. Downgrade my Drobo to 1.3.1 (which previously worked fine with my now-upgraded-and-broken disk pack), create a new disk pack with clean, unseen disks, which would create a 1.3.1 header on the disk pack, and then copy that header off and overwrite the 1.3.5 header that is on my current disk pack disks with the 1.3.1 header, allowing me to put my original disk pack into the physical Drobo, without forcing an upgrade from 1.3.1 to 1.3.5 (which we now know does not work with > 2TB volumes, while 1.3.1 does).

The question is: Where is the firmware header located, and can it be accurately re-deployed from a 1.3.1 disk pack with no data on it, to a 1.3.5 disk pack WITH data on it, non-destructively.

How about buy/borrow a known-good working Drobo and try your disk pack in it?

Definitely think working with a clone of your pack is the safest way. Otherwise any attempt to fix the problem may end up making it worse.

If you have DroboCare, I believe they send you an advance replacement Drobo?

ValerieK indicated to me that even if I put my disk pack in another Drobo with an older firmware, it’s going to upgrade the Drobo to the latest firmware anyway, putting me in the same boat I’m in today.

If my Drobo boots without the disk pack in it, then the problem is probably with the disk pack itself… which (DRI refuses to admit) was trashed during the firmware update from 1.3.1 to 1.3.5.

A full test would include rolling back to the previous firmware to see if it boots there, which would point to a problem with the firmware, and if it doesn’t, then something else is wrong.

Problem is, there is no going back, once that disk pack firmware header is upgraded.

My issue is that this indicates a VERY sloppy QA process. As I mentioned, you NEVER upgrade the data side of any embedded device, unless you’re ABSOLUTELY SURE that the upgrade was successful.

IOW, the firmware is updated, and before the disk is even recognized by the OS, the Drobo upgrades the disk pack to that firmware revision… that’s backwards. They should do it in two steps:

  1. Upgrade firmware, shut down and reboot Drobo

  2. When Drobo boots back up and host OS recognizes the block device, THEN upgrade the disk pack.

There is some subtle incompatibility between 1.3.1 with > 2TB volume and 1.3.5 with > 2TB volume.

Without the ability to roll back to 1.3.1 to validate that, there’s nothing else I can do.

I’m also a bit upset that DRI refuses to even present the “read-only firmware” as an option, so I can get my data off of the physical device itself and start over, nor do they provide me with any other alternatives other than “Oh well, sorry. Format and start over.”

It looks like the Drobo is really just a toy, no different than a single, external USB hard drive. It seems to be targeted at desktop, PC users, definitely not for any data worth keeping, nor for any long-term storage or for workstations or servers. Yes, even the DroboPro would be targeted for desktop users because of these gross limitations.

If you can’t go larger than 2GB, there’s no point in filling it with disks at all, and since DRI doesn’t support Linux, using it there puts your data at risk.

I find it a bit wild that if you update your firmware to a version known to have issues, without the ability to roll back or use THAT version of firmware, you’re stuck, with your data held hostage, until/unless a new firmware revision comes out that can fix the issues caused by the previous version.

How does one take a disk pack and recover data in it, when a bad firmware is delivered, or a firmware upload fails for whatever reason? If your disk pack is updated to contain that firmware revision’s version header, you’re screwed.

And that’s exactly what the Drobo does.

Some additional low-level debugging seems to reveal that my data is indeed intact on the physical disks that make up the disk pack itself.

Granted, each disk, acting as a single drive in an external enclosure will not present the proper partition type to the host OS I’m reading the data from, but I can dd off blocks of data and examine them, and they do contain the data that the Drobo device itself is refusing to allow me to access.

This gives me some hope, that if I can recreate a 1.3.1 disk pack using some spare disks, dd off the identifying header information from the working disk pack to the one that the 1.3.5 firmware trashed, I might be able to downgrade from 1.3.5 (Drobo and disk pack) and retrieve my data again.

The first step is to fully image each of these drives, so I have a solid backup that I can play with in case something goes wrong. That means I need at least 4,500GB of spare disks to store the images upon. Time to run to BestBuy tomorrow and buy some more terabyte drives to add to the pile.

Right now, I’m a bit concerned that DRI refused to give me the read-only firmware they claim is a last-resort measure, used only in cases when it was critical to retrieve the data in a read-only fashion. This does not bode well for their support organization, to leave me hanging out here, high and dry.

I’m going to ask ValerieK to escalate and get me on a call directly with DRI Engineering; not sales and not technical support, but actual engineering. I don’t believe what they’re telling me that my data is just lost, because I used Linux.

My data is there, I just can’t access it through the Drobo, because it’s being held hostage by a disk pack that the Drobo itself upgraded, and can no longer recognize.

ARGH!

I followed up on your case. Since you are in an unsupported configuration (i.e. >2tb Linux, Drobo Gen 2), the ‘read only’ image will not resolve your problem. If we believed the ‘read only’ image were going to work this would have been sent to you.

Hacker, you kept saying your data is there but is it really there when you had not even exercise good backup protocol?

Backing up data involves a lot of procedures. A good website is http://www.taobackup.com and gives you insights of what it takes – yes TAKES – to have a good backup system. Personally, you’re not even close.

I had an experience once of a client who was desperate like yourself, unwilling to relent to the possibility that the data was unrecoverable. He was just like yourself, having an unsupported file format on a RAID system that had warned him NOT to use the file format greater that what was recommended. Of course, he thought because it was working before hand, he thoroughly believed the data he stored on his RAID setup was good. He did eventually paid a lot of money to get the data back. Sadly though, his data was corrupted! It was not corrupted due to the recovery. It was corrupted because it was already corrupted when he decided to use a file format that was not supported in the first place! He did not even do an integrity check on all his data, therefore he wouldn’t even know that his 2TB of photos were already in the process of total corruption as he kept saving and saving to the RAID box. He didn’t listen to me. He spent more than $10,000 just to recover junk, though few photos were useful.

Hacker, unless you did an integrity check by restoring your 1.2Tb backup back onto a 2Tb drive and checking the validity of the restoration itself, how would you know your data so to speak is good data!?!? If you did a restore back onto a 2Tb drive, you would have a backup of your Drobo. But you don’t.

You are playing a mind game really of something you think is there, but isn’t. I know you don’t want to loose 300Gb of your precious life stored, but at taobackup.com, others had lost MORE since the late 90s. Where were you then?

Don’t blame Drobo. Even the best backup software made in the world has problems retrieving DAT tapes backed up from older software and was made worse by corrupting the tape during restoration. It happens all the time.

Hopefully though, you will learn from this mistake and do proper backups. Many people move on. It seemed you are unwilling to do so.

[quote=“DumbTechDude, post:8, topic:948”]
Backing up data involves a lot of procedures. A good website is http://www.taobackup.com and gives you insights of what it takes – yes TAKES – to have a good backup system. Personally, you’re not even close.[/quote]

You’re preaching to the choir, and the person who carved the pews. It’s fine, get it out if it makes you feel better.

[quote=“DumbTechDude, post:8, topic:948”]
Hacker, unless you did an integrity check by restoring your 1.2Tb backup back onto a 2Tb drive and checking the validity of the restoration itself, how would you know your data so to speak is good data!?!? If you did a restore back onto a 2Tb drive, you would have a backup of your Drobo. But you don’t.[/quote]

You didn’t read, apparently, but that’s common for those who attempt to speak, before they attempt to listen. Your username seems very apropos in this case.

Before I store anything on any backup medium (in this case, the Drobo’s disk pack), I run a full md5sum check of the source, fsck the target (took 18 hours on the Drobo), and then back up the data to the Drobo, and execute a second md5sum check (assisted by the md4 built into rsync), to verify that what was on the target, reached the destination, and was written to known-good inodes.

[quote=“DumbTechDude, post:8, topic:948”]
You are playing a mind game really of something you think is there, but isn’t. I know you don’t want to loose 300Gb of your precious life stored, but at taobackup.com, others had lost MORE since the late 90s. Where were you then?[/quote]

I know it’s there, because I can read it and see it… I just can’t (yet) reassemble all of it, because of the way it’s layered onto the physical drives. Unless the Drobo corrupted the higher-level filesystem itself (i.e. wrote to the data areas of the platters), during the firmware upgrade and subsequent power-on/power-off cycles as suggested by DRI, the data hit the platters intact, and should remain so.

[quote=“DumbTechDude, post:8, topic:948”]
Don’t blame Drobo. Even the best backup software made in the world has problems retrieving DAT tapes backed up from older software and was made worse by corrupting the tape during restoration. It happens all the time.[/quote]

You’re off on some other tangent here. I think you missed the point, so let me recap so you don’t miss it this time:

1.) fsck the filesystem on the Drobo (18-hour operation)
2.) back up incremental data to Drobo (I’ve successfully done this about 8,700 times in the last year on this Drobo using the 1.3.1 firmware with this 8TB volume)
3.) attempt to delete older snapshots from Drobo (at extremely-degraded speeds)
4.) safely shut down Drobo
5.) power up Drobo and upgrade firmware from 1.3.1 to 1.3.5 using approved tools, at the suggestion of DRI and others in the forum to alleviate performance issues
6.) safely shut down Drobo to affect the firmware upgrade
7.) dead Drobo, as it never completed booting after that upgrade

How is this in any way related to my backup or restore methodology again? I think I’m missing the point. DRI seems to be missing the issue as well, blaming my volume size as the culpret.

It could be an 8TB HFS+ or 8TB NTFS volume, but it doesn’t matter. Drobo itself can’t address the disk pack, and refuses to stat the drives to complete the boot process. That happens well before the filesystem sitting on those drives is even in the picture.

[quote=“DumbTechDude, post:8, topic:948”]
Hopefully though, you will learn from this mistake and do proper backups. Many people move on. It seemed you are unwilling to do so.[/quote]

Incorrect. I’m unwilling to accept excuses and ineptitude in the place of actual solutions. There’s a difference.

The good news is that I’ve gotten quite a bit further along than anyone here or at DRI support believes is possible. Your opinion of the matter, means nothing.

Move along…

Wow. I’m sure help will just spill forth from this point forward, hacker.

[quote=“Da_Man, post:10, topic:948”]
Wow. I’m sure help will just spill forth from this point forward, hacker.[/quote]

I think I’m well past that point with DRI support.

I asked them to schedule a call between myself and DRI Engineering, so we can all start talking the same language. I’m waiting to hear back on that from them.

As far as “help” from DumbTechDude, I could care less, he’s just ranting and self-pontificating. That’s just background noise at this point.

Okay, I’m confused again. Is Drobo your backup storage, online storage medium, or both?

i thought it was the primary and only storage for a lot of his photos from what he was saying earlier???

but now im confused by him talking about using it for backup - surely that means the originals are somewhere else?

My Drobo was attached to a server as a physical drive, serving up mirrors of CPAN, Gutenberg, Cygwin, etc. to the community.

It was also holding backups of all of my internal systems, managed via rsnapshot and rsync on Windows and Linux (hourly snapshots). I also maintained my digital photo archive there (.jpg and .CR2 files, going back about 10-11 years).

I also manage my music collection from the device, via Ampache.

I also had several network drives mounted from Linux and Windows machines, pointing to the Drobo (using sshfs), where my documents were kept (current and archived).

So the answer to your question is, “Yes, all of the above.”

Okay, thanks for clarifying.

So at least some of the data being backed up was being backed up to the same physical device (Drobo), and hence when you lost access to the Drobo, both the data and its backups became inaccessible.

So DumbTechDude said you don’t have backups, and you say you did.
You’re both right, because there’s a difference there.

DumbTechDude’s definition of a backup is a completely independent backup, one that you can pick up and walk away with or store off-site. It can be read elsewhere, with no overlap in requirements between the source data medium and the backup data medium. In other words, a single point of failure in the system should not make the both the source data and its backups inaccessible.

In the real world, this could mean be another separate Drobo that stores the backup images, a portable hard drive, online storage, or something else. The key is that the backup data is readable and usable independently from the system that stores the source data.

Unfortunately your backups (unless they were stored/copied elsewhere), aren’t completely portable, because they are on the same Drobo that the source data was on. That’s not to say that source-local backups aren’t useful, they are - Restore Points in Windows are stored on the system drive for “rolling back” to a known-good system state, and local snapshots make it quicker to recover a deleted file or previous version of it.

I realize this doesn’t help the current situation, and I still sincerely hope that you can get your data back. Just trying to shed light on the other side.

[quote=“hacker, post:9, topic:948”]

[quote=“DumbTechDude, post:8, topic:948”]
Backing up data involves a lot of procedures. A good website is http://www.taobackup.com and gives you insights of what it takes – yes TAKES – to have a good backup system. Personally, you’re not even close.[/quote]

You’re preaching to the choir, and the person who carved the pews. It’s fine, get it out if it makes you feel better.

[quote=“DumbTechDude, post:8, topic:948”]
Hacker, unless you did an integrity check by restoring your 1.2Tb backup back onto a 2Tb drive and checking the validity of the restoration itself, how would you know your data so to speak is good data!?!? If you did a restore back onto a 2Tb drive, you would have a backup of your Drobo. But you don’t.[/quote]

You didn’t read, apparently, but that’s common for those who attempt to speak, before they attempt to listen. Your username seems very apropos in this case.

Before I store anything on any backup medium (in this case, the Drobo’s disk pack), I run a full md5sum check of the source, fsck the target (took 18 hours on the Drobo), and then back up the data to the Drobo, and execute a second md5sum check (assisted by the md4 built into rsync), to verify that what was on the target, reached the destination, and was written to known-good inodes.

I say:

Against what Hacker? This presumes you have a perfect master stored in a certified file format. By reading your recent posting, you didn’t or otherwise we wouldn’t be having this discussion. The problem with your flawed theory is that, you are using an unsupported file format storing your precious data and assuming that by doing CRC and fault checks
guaranteed your data to be sound and secure is perfectly acceptable to you? This is no different than someone attending university, administering his own tests and evaluating his own performance by himself and then granting himself a degree in the process, without going through a certification governing body. I know people spend days verifying a copy of a copy thinking they are smart. It’s like the Dumb verifying the Dumber. Am I right on this?

A master copy is free from virus, malware, file corruption and is saved in a format that is retreivable with approved means based on the supported hardware. You have one of those?

[quote=“DumbTechDude, post:8, topic:948”]
You are playing a mind game really of something you think is there, but isn’t. I know you don’t want to loose 300Gb of your precious life stored, but at taobackup.com, others had lost MORE since the late 90s. Where were you then?[/quote]

I know it’s there, because I can read it and see it… I just can’t (yet) reassemble all of it, because of the way it’s layered onto the physical drives. Unless the Drobo corrupted the higher-level filesystem itself (i.e. wrote to the data areas of the platters), during the firmware upgrade and subsequent power-on/power-off cycles as suggested by DRI, the data hit the platters intact, and should remain so.

I say:

Let’s be clear the difference between being able to see 1s and 0s than seeing actual data. You know what, I see 1s and 0s in a failed RAID, but are those 1s and 0s the same as your master copy is the question. You assumed it is, but I suspect since you do not possess a master copy to compare with, how can you be sure?

You said:

1.) fsck the filesystem on the Drobo (18-hour operation)
2.) back up incremental data to Drobo (I’ve successfully done this about 8,700 times in the last year on this Drobo using the 1.3.1 firmware with this 8TB volume)
3.) attempt to delete older snapshots from Drobo (at extremely-degraded speeds)
4.) safely shut down Drobo
5.) power up Drobo and upgrade firmware from 1.3.1 to 1.3.5 using approved tools, at the suggestion of DRI and others in the forum to alleviate performance issues
6.) safely shut down Drobo to affect the firmware upgrade
7.) dead Drobo, as it never completed booting after that upgrade

How is this in any way related to my backup or restore methodology again? I think I’m missing the point. DRI seems to be missing the issue as well, blaming my volume size as the culpret.

I say:

It is related, because what you are actually doing is basically appending backup ontop of a backup without any means of testing and doing integrity checks on the actual integrity of the backup itself against a master. You are fully aware that you are saving to an unsupported file format, which means that there is a chance of data loss would it not? Securing a master copy saved in an approved format would be what an intelligent person would do wouldn’t it? Again, if you have a master to compare with, we wouldn’t be discussing this.

Hacker, you can spend all the time you want trying to recover the 1s and 0s you believe you saw in your Drobo. This is your right and yes, it’s your time. Personally, I do not see the reason why you should keep pestering DRI that it’s all their fault. I personally think it’s not. They provided the warnings, including in their firmware page that you must ensure to have a backup before performing the upgrade. I thought that warning was very clear. Jennifer was clear that a read only firmware will not solve your problem because of the unsupport file format you used, which should mean something to you when you attempt the recovery.

Again, I really feel for you for the lost of data. But perhaps moving forward, you would ask for help but refrain from blaming DRI?
And if you do succeed in retreiving your data, don’t you thnk this would become a business opportunity as an expert in Drobo recovery?? You can make a good living helping others who I’m sure will suffer the same fate as yours. You do like money do you? Perhaps rather than burning bridges now, you might want to foster some good relationship with DRI and the rest of the Drobo community.

The door swings both ways.

Cheers,

My suggestion.

Cheers,

[quote=“DumbTechDude, post:16, topic:948”]
Securing a master copy saved in an approved format would be what an intelligent person would do wouldn’t it? Again, if you have a master to compare with, we wouldn’t be discussing this.[/quote]

When the “supported” NTFS and HFS+ core filesystems are updated to support the file naming standards, conventions and ACLs used and supported on ext2, ext3, xfs, jfs, and so on… I’ll be more than happy to test a migration. Right now, I can’t put my data on NTFS or HFS+, because it can’t support it. Hence, I’m using an industry-standard, supported, decades-tested, ext2/ext3 filesystem to store that data.

The read-only firmware allows the disk pack to be mounted at all costs. Since the Drobo can’t complete the boot, the disk pack is the problem. Since the disk pack was upgraded by the Drobo firmware upgrade, and not by the host OS, then DRI and its firmware and/or firmware upgrade methodologies are directly at fault here.

This has absolutely nothing to do with the filesystem that sits on top of the physical disks, because the Drobo itself, can’t even recognize the physical disks as a single unit, after that upgrade. This disk pack could have been HFS+ or NTFS, and it wouldn’t matter. The point is, the Drobo hasn’t even handed the block-level device over yet, so the filesystem that sits on it, is irrelevant.

Let’s go through that again:

  1. Several hundred reboots under 1.3.1, without a single failure
  2. Upgrade to 1.3.5
  3. First boot of Drobo under 1.3.5 results in failure

How is this related to me, my process or the filesystem? I’m failing to see your point.

Actually, no. Once I get my data off of the Drobo, I’m ejecting it from my inventory, and will be strongly recommending that others do the same, unless you’re on a legacy platform like Windows and/or Mac, and use it for temporary storage of ‘cached’ data that exists elsewhere (i.e. the flexcache analogy I mentioned earlier in another thread).

Money means nothing to me, so no… that wouldn’t be my motivation. I would be motivated to create a better, more-open system that is a bit more impervious to failures of this magnitude, however. Not being able to downgrade, not being able to read the disk pack outside of the physical device, not being able to mount the volume as read-only under this or any other environment are some pretty significant design flaws, IMHO.

Bhiga,

Thank you for adding some comments to my post to Hacker, though Mr. Hacker is having a difficult time reading and understanding my posts about backup.

The problem I see with running an unsupported format, Drobo not withstanding, is you don’t really how and what the Drobo is doing to thestored data. Since Drobo is virtualizing space that Hacker doesn’t have now but will in the coming future, that data is stored in a way that DRI feels unsafe. Which is why running all kinds of data checks are useless, which was what I was trying to tell Hacker. He failed to listen because I had seen this many many times and he thinks he is right.

By the way, backing up data onto the same storage unit eventhough it’s partitioned into different section is NOT backup. You can create 3 partitions as seperate drives on the same Drobo, but that’s not backup because it is stored on the same unit and laid out on the same unsupported format. My last post was about a tier 1 backup, a basic backup setup. There are a lot of people like Hacker who has a Drobo but need something else to backup on. I usually recommend the Mediasonic Pro Raid 4Bay or its sister Mediasonic Pro 4 Bay if budget is tight and the client can not afford another Drobo. In the case of the Mediasonic 4 Bay non-RAID version, you get 4 individual drives and 4 individual drive letters. eSATA interface is faster than regular USB from Drobo gen 2! If you back up a Drobo to the Mediasonic drive, this is tier 1; Drobo would be master and Mediasonic would be the backup. But there is still a problem with the master, because with disk imaging software or snapshot roll back, it images the data well included its flawed file format too! With Acronis True Image which I use often, if you image a FAT 32 drive to a backup and then say format that FAT 32 drive to NTFS and the restore it from a Drobo. Guess what, the NTFS drive becomes a FAT 32 with all its flaws that were imaged from the master – this is to be expected! Again, we do data restoration on all our servers and computers. We are not afraid to wipe out any computer and then am 100% confident that the restore will work!
Had hacker done this? Have any one in this forum done this with 100% confidence?!?

Otherwise, what’s the point of having a backup not knowing that you can restore it back with non-corrupt full files.

That was the thesis of my post and had no intention to lay any blame.

DTD.

[quote=“DumbTechDude, post:18, topic:948”]
Otherwise, what’s the point of having a backup not knowing that you can restore it back with non-corrupt full files.[/quote]

It seems like you’re still stuck on this point…

As I’ve alluded to (and probably not specifically stated outright, which I guess you’re after), the data is (was?) 100% intact, because I’ve used that same data hundreds of times to reproduce rebuilt systems, restored hundreds of gigabytes of documents, images, VMware images, and public mirrors of very popular OSS projects.

If the data was corrupt in any way, I’d know about it pretty quickly, and be getting hundreds of emails about it every day.

[quote=“hacker, post:19, topic:948”]

So you “DO” have backups on those systems that you had restored?

See I’m confused here. First you said you lost all of your data. Now, you said you restored data on your Drobo hundreds of times to rebuilt systems which means you must have those data somewhere else in more than a hundred machines?

So what is your problem then?

DTD.