Drobo

Files Appear Corrupted on Drobo

We are a photography business and recently purchased the Drobo S for our small business, backing up primarily Lightroom files, raw images and jpegs on a Mac Pro (OS 10.6.4, 2x3 GHz Dual-Core Intel Xeon, 8GB RAM). We have 3-2GB drives in the Drobo, connected through USB. We backup nightly using Drobo Copy.

Recently, we have noticed that some of the files that were backed up using this software appear corrupted on the drive. For example, the preview image shows some JPEGS with parts of the image grey or repeated:


Example 1


Example 2

When we open the file, it shows the same image with part of the image grey or repeated:


Shot at 2010-11-14

This occurs across several different folders. Sometimes, the file won’t even open, stating that data on the image is missing.

I have noticed that sometimes, if you wait a few seconds (click one file, then click another, then click back to the image in question), the error will fix itself and the file (and preview) will open fine. In other cases, they appear to be permanent, and nothing I do will fix them.

I replaced the USB cable - no difference.
I ran Disk Utility > Verify Disk - it says it’s fine.
I ran Disk Utility > Repair Disk - on a few instances, it ran correctly. Today, I ran it again, and receive the following errors that it cannot mount:

Thoughts?

I’m a Windows guy so I can’t point to a particular program but what you need is a program to create and verify MD5 or SHA1 hashes. Or switch to a copy program that does a full CRC verify.

Verification of the MD5 or SHA1 hash (or CRC verify) will ensure that the backup file is data-wise identical to the original file. If the original and backup hashes don’t match, the copy was not completely successfully.

Cool man, boot from your OS X DVD and try and run the repair again from there.

When you open the pictures in Preview or Photoshop as apposed to ‘QuickView’ do the files remain corrupted?

Lastly, are you connected via USB or Firewire?

If you have a copy of Disk Warrior, you can run that as well on the volume.

hi just to mention, probbaly pc only (but maybe command line/mac versions out there) but theres a good tool called quickpar which is great for parity checking

I use QuickSFV for file verification. Search for “SFV” to find other tools that you may like better.

I experienced similar problems when I had a faulty RAM in my PC. You should verify that your computer is OK and/or try it from another computer.

Highly recommend Memtest86+ though I’m not sure it works on all Macs.

This won’t help the OP since it’s Windows-only, but the easiest checksumming software that I’ve found is Checksum from Corz ( http://corz.org/windows/software/checksum/ ). Once it’s installed I can right-click on a folder, choose “Create checksums” and it’ll cruise through that folder structure and create a hash file in each folder that contains an MD5 value for each file in that folder. I can then later right-click that folder and choose “Verify checksums” and it’ll cruise through and verify all the files against the checksums. Or, I can go to a particular folder and double-click on the hash file and it’ll just verify the files in that folder. When done it’ll tell me that everything matched, or give me a list of files that didn’t match. It’ll ask if you want to save the list to a file (which is formatted in HTML), which usually I do since it gives the complete paths to all mismatches or missing files.

It’s very simple, very easy, and oh so handy. I back up from my main PC to a RAID on a different PC, and then from that to my Drobo. I compute the checksums on my PC before copying the files, and then verify the files after they land on the Drobo. If the checksums agree with the files on the Drobo then I know that the copy on the intermediate RAID is good too, so there’s no need to verify the checksums there.

It also provides me peace of mind when doing a relayout on the Drobo. When I do any activity that involves a relayout I verify the files against their checksums afterward so that I KNOW that nothing was corrupted. So far the Drobo has been rock solid and never corrupted anything, but you should never trust anything 100% when it comes to your valuable data. I just right-click on the root folder where my backups are, choose “Verify checksums”, and let it do it’s thing for however long it takes.

I do the same thing with files on the flashdrive that I keep with me at all times. A few weeks ago I did a verify on it and found that 2 files were corrupted and I was able to restore them from a backup. I don’t know for sure how they got corrupted, though I do recall that there was an unexpected reboot on my PC a month or so ago while it was attached, so maybe something got hosed because of that. It’s possible that the flashdrive itself might be corrupting data, so I’ll be testing it occasionally and will replace if necessary. The point is, I wouldn’t have known about the corruption if I hadn’t been using checksums.

For flash drives in Windows - use ChkFlsh aka Check Flash

Yes!!! I have the same problem. Some image files are corrupt also large movie files (8gb or so).

I could use a checksum copy app, but can anyone say conclusively if the data is safe once on the drobo?

At least the checksums would help identify changed files. From there you can try to determine what the cause of the corruption is.

I’ve had random corruption happen due to network caching (this was pre-Drobo), but that was back in the 2000/XP days…

Most recently (again, non-Drobo) I copied a file from my desktop to a USB stick, Safely Eject-ed the stick, removed it, took it to my wife’s computer, copied the file to her desktop, then launched it (it was an installer) - and it gave an error.

Ejected the stick from her machine and did the same thing again, and it was fine the second time.

Unfortunately I didn’t stop to check whether the copy to her machine was corrupt or the copy on the stick was corrupt though.

Point is, as DrBunsen pointed out as well, bad copies can and do happen.

If you are having corruption, first I would check that the filesystem is intact (chkdsk, disk repair or fsck - depending on OS).

hmm i liked the sound of that checksums tool drbunsen said but the page gives a 404 not found :frowning:
i remember Suite B asking for a drobo tool to force drobo to access all files, (so that it’s in-built drobo parity verification and corrections methods etc would trigger) maybe that checksums can do it but either way the link is down.[hr]
btw, i also had some corrupt images on another (internal ide drive), i copied them to zipdisk and tried using lots of free tools to recover the rest of the picture… i managed to get a few milimeters more but apparently compressed formats are done in such a way that i dont think you can

i currenlty have those images stored on the drobo (to help preserve whatever is there) and so far, not a single image/wallpaper, clip art or any of my own creations on the drobo have never become corrupt, (thats since 2008 so far) - at least as far as ive noticed… hmm i wish that checksums was around - saves me having to create quickpar par2’s per folder :)[hr]
i found it :slight_smile:
http://corz.org/windows/software/checksum/

I use ExactFile, guess it depends how fancy/GUI you want, though ExactFile also has a command line version. :slight_smile:

That Checksum tool looks nice too though, I might start using both for greater peace of mind!
BRAIN WARNING: Paranoia level has increased to orange. Stop before it gets too stupid. :smiley:

thanks for exactfile (the brandon there isnt you by chance is it?) :slight_smile:

No, different Brandon… ExactFile’s Brandon is far more… disciplined. :smiley:

ACK! My apologies - I didn’t notice the incorrect link in my earlier post. The forum software included the “).” as part of the URL. (Paul got it right!)

Data integrity is highly important to me. I really REALLY wish that NTFS included checksum information for each file in the file system entries. Just a simple CRC32 would be fine. I would love to be able to use something like ZFS in Windows since it includes checksum information, but until that happens I’m stuck with external checksumming tools.

I downloaded and looked at ExactFile, but the way that it works is a bit restrictive for me. I like the Explorer integration of Checksum - it’s nice to be able to right-click on a folder or drive letter in Explorer and choose “Create checksums”. If I’m wanting to make checksums I’m invariably already working within Explorer. With ExactFile you have to separately start ExactFile. At that point you can either drill down to the folder you want to start checksumming from, or you can drag the folder from Explorer to the input line in ExactFile (which is the way that I’d do it).

However, there are a few big things that I don’t like about almost all checksumming programs, including ExactFile.

  1. Let’s say you want to create checksums for your D: drive. With ExactFile (as with most other checksum programs) all the checksums for the drive would then get put into a single file in the root of the D: drive. Let’s say a year later you then want to burn the contents of D:\Media\Pictures to a DVD. There isn’t a convenient way to first make sure that the pictures in that folder are still good via checksums, and to have checksums follow the pictures to the DVD. Since the checksums that were calculated a year before were for the entire D: drive you’d have to verify the whole D: drive, to verify the contents of just that folder. I’m assuming that you could cancel the verify after it reaches that folder, but that could be a long time. You could work around that by originally creating a separate checksum digest for each folder off the root of D:, which is how I used to do it with earlier checksumming tools, but if I have D:\Media\Movies then I could be verifying for hours before I get to D:\Media\Pictures. The upshot is that unless you happened to have used ExactFile to create a checksum digest of just that folder, you’re going to have to verify more files than what is needed. Plus, you’ll need to create a digest for that folder in order to move it to DVD. It’s all extra work and extra time.

  2. Then there’s the problem with changes to the file system structure. If you make a checksum file of your D: drive and move folders around, that checksum file is now going to need to be totally redone, or edited by hand, to reflect those changes. I move folders a lot, so that would hurt me a lot with ExactFile’s method.

  3. Then there’s an issue with file exclusions. There are some files that I know I’ll be changing fairly often. Because of that, a checksum that I make of it today will not be any good a year, or even a week, from now. I couldn’t see any way to tell ExactFile to not create checksums for particular filenames or extensions.

All the above things are handled by the Checksum program from corz:

  1. When Checksum creates hashes (checksums) for all the files on the D: drive, it stores the results in separate .hash files, one in each folder. Each hash file contains the hashes of each file in that folder, and only that folder, and the entry for each file is just the file name, no absolute or relative path is included. So if you want to ensure that the pictures in D:\Media\Pictures are OK, you would double-click on the file named Pictures.hash inside that folder. Checksum would then verify that the checksums in the hash file match the associated files in that folder. If you had subfolders you could right-click on the folder, choose “Verify checksums”, and Checksum would verify the checksums in all the hash files down that part of the folder structure. Once you’ve verified the files against the checksums in that folder (or folder structure) you can burn that folder or folder structure to DVD, and then you could check the hashes on the DVD to see if it was burned properly.

But… what if there were new files added to that folder since the .hash file was created? When you tell Checksum to create checksums on that folder where it had already been run before, it will find the existing Pictures.hash. It will not recalculate the checksums for the entries that are already in the .hash file, but will do so on the new files and append the information to the end of the existing .hash file. You won’t know if the new pictures were corrupted from the time they were copied to the time you make the checksums, but at least you’ll know from then on, and you won’t lose the original checksums from the year before.

  1. If you move the Pictures folder to a different location then there isn’t any problem since the hash file doesn’t care about relative or absolute paths. Pictures.hash doesn’t care if it’s in D:\Media\Pictures, or Z:\folderA\folderB\folderC\Pictures.

  2. By editing its .ini file I can tell it to not run against particular filenames, or for files with a particular extension, or files in particular folders. So I can avoid a lot of false negatives by telling it to not hash files that I don’t care about (e.g. .ini, .txt, .log, .lnk)

A few gotchas: If you rename a folder (e.g. Pictures to Photos) you need to try to remember to rename the .hash file (Pictures.hash to Photos.hash). If you’re verifying checksums then Checksum will look for every .hash file in a folder and verify the checksums against the files in that folder, so it’ll find Pictures.hash and verify the files. But if you then tell Checksum to create checksums, then it’ll create a new file called Photos.hash and calculate checksums on all the files again, giving you 2 different .hash files in that folder. Then if you tell Checksum to verify checksums on that folder it’ll do it for both .hash files, so it’ll take twice as long. Another gotcha is that if you delete one or more files from the folder then when you tell Checksum to verify checksums it’ll report it/them as missing. So if I delete some files from a folder, and that folder contains a lot of large files that would take a while to checksum, I’ll save time by editing the .hash file and manually removing the entries. If they’re in a folder with files that’ll checksum quickly then I’ll delete the hash file (after I’ve verified that the other data is OK) and recreate the checksums.

But remember that most such gotchas are true for all the checksumming programs. Until checksuming gets built into the Windows file system, Checksum is about as perfect as I’ll probably find.

Please note that I did find a bug in Checksum a couple years ago. It doesn’t want to properly handle files/folders with characters that aren’t plain 7bit ascii, and if I recall correctly it would recreate the entry for that file in the hash file each time it was run. So a file named Noël.mp3 would get done multiple times due to the special character ë. I reported it and the guy came up with a fix that he sent me, but for some reason he hasn’t updated the version that he has on his site with that version. Hopefully he’ll release that soon. In the meantime I wouldn’t mind sharing the version I have if you’d like it.

Ray

ExactFile can create single-file checksums too, though I agree with you that Checksum’s shell integration is a huge plus! :slight_smile:

…and I too wish that checksums were included in the filesystem itself. Would make a lot cleaner.

I’d appreciate if you could share the fixed version of Checksum that you have!

thanks ray,
i think exact file is still not even version 1 (with a lot more to come) but yes i think checksums is better at the moment, though i like exact files options to obtain / generate a particular method in particular in case needed,

i use checksums integration (quite often today actually) when going through my (old media recovery procedures) LOL - even have a separate thread :slight_smile:

http://www.drobospace.com/forums/showthread.php?tid=1999

(btw if you want to share your tweaked version, you can attach it to your profile, and then reference it as an attachment when you later post a reply. thats how i embedded my dvdisaster pics)