Serious Backup data-life strategy considerations?

Hi guys, i find myself in the following situation, and not too sure what is a good

way forward…

(HISTORY)
over the years, ive ended up with quite a bit of “data”, on various storage media.

Data is quite diverse:
ranging from family photos, document scans, large multi-versioned multimedia

projects, family video conversions from cinefilm, games development content, anims,

backups, sound and graphic icons/librarys etc. (small files/larger files)
(even huge collections of doom2 wad files and game mods - which even now there are

massive websites doing well for that niche) :slight_smile:

Storage mediums:
a) on old ida/pata drives (still working but not sure for how long) (15yrs)
b) on older cd’s (a good few hundred or so main ones) (10yrs)
c) on data dvds around 100 (last 5yrs) (mainly compressed data)
d) on cd/rw (since approx 10yrs)
e) on iomega zip disks (10-15yrs approx)

Total data probably about 1tb compressed to (estimate 1:10 ratio) say 10tb uncompressed).

(heh, have about 100 of these - back in the day, friends often would offer me cheap

Mac formatted ziptools disks (macs werent nowear as popular as pcs back then) for a

few quid… this was excellent for me as pc formatted ones were ranging from about

£4 in massive bulk deals, to around £11-15 for a 1 off) - ended up getting them to

go through all the departments buying up old mac tools disks, and i simply re-formatted them and they were perfect LOL - got super deals

lord sugar would be proud of :smiley:

Problem:
while i probably have more data then i “actually” really need if i was to sift

through it all - i dont have super amounts of time to do that and just want to

ensure i can still access that data in the future.

ive heard (from Suite b i think) that the drives only last for about 20 years, and

on various websites that different cd dye materials can only last so long, before

they simply dont work anymore) -

buzzwords include short life strategy, metal azo, something cythaninene, long life

but dont use much, or use as much as you want but only for 5 years etc…

Solutions?:
on the zip disks, ive been using TIP from Steve Gibson (grc)
that seems to work quite well, at only about 1min per zip, taking around 2 hours

every 3 months or so.

on the drives, ive started (dumping) folder contents onto the drobo (for data

rather than installed program files etc)

on the cds/dvd (optical media), am not sure if i should just dump it all onto a

drobo, or if its quicker, (or more cost effective) to buy or rent a basic cd

duplicator, and to just buy a set of cheap blanks and basically clone all the old

ones onto new ones?

unless ive made a mistake, even for standard level brands, the cost per GigaByte

seems to still be cheaper than hard drives. eg 33p/gb hd, vs 10p/GB on a good dvd

say.
but they needing another whole shelf to put them on lol :frowning:

or is it better to pay a hosting company each month and get some sort of

protected/backed up area with encypted tunnel to be able to access it when needed

(not sure how much that wouldcost though)

Peter B Clements (of quickpar fame) had good info and tools (before his site got done by tukish hackers) with people using some kind of universal base version of an atapi cd reader (which would apparently always create the same byte-level cd extracted image data, upon every runtime), which was used to create a master parity quickpar set, and in the event of a scratched disk it would be used to read each and every possible bit of info, and then when used with a set of par2 data you could effectively re-create an iso image to then make a fixed new cd. (that was the theory but i cant seem to find more about it recently)

i dont know how many of you are in the same boat - but i guess as time goes on we’re all collecting or generating more data each year that theres probably a lot of you in a similar boat? :slight_smile:

always good to hear your thoughts
thanks,

"seems to still be cheaper than hard drives. eg 33p/gb hd, vs 10p/GB on a good dvd "

check your maths:

2,000GB WD drive from PC world 80 quid

thats 4p/GB

hard drives and now the cheapest and fastest form of storage (large scale) storage around

oops i sit corrected :slight_smile:

btw i found a good tool which might compliment things…
its called “dvdisaster”

i can put my cds in, and check if all the sectores are readable, and if any show yellow or red, i’ll dump them to drobo.
that should help provide a halfway solution for the time being

Ahh, the good 'ole Zip disks and Click of Death… At one time I had all the Zip100 models - SCSI, Parallel, ZipPlus and ATAPI…

I can tell you from first-hand experience that things do have a lifetime.
CDs and DVDs - they’ll wear by either light or humidity.

I went back to my parents’ house to collect my “old” data on Zip, floppies (5.25" and 3.5"), CD-Rs and CD-RWs…
Most of the data was at least 10 years old.
A lot of the CDs were unreadable.
Some burned CDs had succumbed to what I guess is dye degradation
Others had visible oxidation “holes” because the edge seals had degraded (in the laserdisc world, this is called “laser rot”)
The Zip disks were mostly OK, except for a few.
The floppies, the oldest of the types of data, were hit-or-miss. Granted, they were 10-15 years old at this point but some of them literally broke in the floppy drive - the floppy head wore a hole through a few.
Others just had trouble reading.

Guess I’ll never get to relive my old ANSI animations in TheDraw… :\

So, end of semi-sad story…

For your optical media especially, I would look to imaging the discs - you can automate utilities like ImgBurn

dvdisaster looks interesting, but if I’m understanding it correctly, it creates discs with error-correction data, which is good, but to protect what you already have you’d have to read it and create new copies with dvdisaster anyway, which IMO would take longer than simply pulling the content to your Drobo?

Then again, this is from someone who has a CD/DVD autoloader…
But a Primera Composer isn’t too much money nowadays on eBay, or you could spend a lot more and get a Nimbie type unit. I retrofitted a SCSI Composer with a SATA-USB bridge board and a new drive for under $200 (not including price of the Composer unit).

Where are you located? If you’re nearby we could figure something out - at least for your optical discs.

For floppies I use WinImage (a lot of the self-extracting boot disks use WinImage) but I don’t know of a floppy autoloader. Maybe you can find a crazy person with a USB Floppy RAID
Seriously though, $20-$30 for a USB floppy drive isn’t much of an investment if it’s something you really want to keep.

This one’s more of a diligence thing - stick it in, let it image, change discs, just make it a habit while you’re on the computer or something. Same with Zip disks.

You can always mount and look through the images later. The point would be just to get your old data in a less-perishable state, not to sift through it all - unless you have more data than you can currently store.

Data-longevity’s candle burns at both ends - one for the media readability, and the other for the availability of the hardware to read it. Finding a well-aligned 1.2MB 5.25-inch drive is… not fun.
Sorcerian did not read well. Still hoping for a Sierra On-Line “Everything” collection one day. I lost, then found, then re-lost my The Wizard & The Princess floppy long ago, in a galaxy far, far away. I can see it in my mind’s eye still… nice black floppy with brown and beige label… and the snake early in the game. Darn snake! sniff

thanks for the info Brandon, am checking out imgbrn.

dvdisaster appears to have 2 modes… to create error correction ecc data and burn to the cd at the time of burning.
or, (if the cd is still readable) to create an ecc file somewhere else (like a drobo), which can then be used in conjunction with the cd in future. (eg say after 3 years, 100 sectors are playing up, you can use all the other secotors plus the ecc file to create another iso of your cd (which can then be mounted or reburned etc)

that memory stick and floppy raid looks cool. i wouldnt go that far myself - but i did used to have at least 3 Amiga 500 floppy drives all daisy chained, so that i wouldnt have to keep inserting disk x all the time :slight_smile:

that cd autoloader looks interesting but also serious money, i think what you said about imaging them could work.
(maybe better than copying and pasting all, as no files to virus scan or take more space) and i could probably enable WinXP ntfs compression on the iso folder, and simply mount them when needed etc.

i got the maths wrong earlier though as docchris pointed out (i blame the “Caliber beer” i tell you) :smiley:
it actually looks a lot more attractive now using data on drives, rather then shelfs of cd boxes etc

ive got 7x1.5tb WD15EADS’s on the way - found a bargain £44
so i think i’ll have to start going through the boxes at random to see if i can see which types of discs or brands seem to still be fully readable with all sectors etc, and then to start dumping things before laser rot kicks in :slight_smile:

(interestingly enough… the older media seems to be more branded stuff like Sony and TDK/Verbatim/Memorex - its the newer media which is the cheaper unbranded ones - so maybe we do get what we pay for) :slight_smile:

Sorcerian looks nice btw: looks a bit like Super wonderboy in monsterland, and final fantasy 2/4 (heres a link for you)
http://www.hardcoregaming101.net/sorcerian/sorcerian.htm :slight_smile:

Hmm… I like the “add ECC data to the disc” concept, but it still feels a little moot. It’s kind of adding duct tape to the splintered pole rather than replacing it.

As I often tell customers - “You pay either way - in time or in money.” Adding the ECC data would buy you more time, assuming the discs can still have sessions added. Just spreads the work out over a longer period of time, I guess. Better than nothing I guess, but I’d still go with the imaging+verify route if you can.

Seeing as your currency is in £, there’s an ocean between us, so I won’t be of much help on imaging your discs. I wouldn’t be comfortable asking someone to ship their only copy of data overseas.

CD-Rs were funny in the beginning. I had more green TDK’s that read fine compared to gold, silver and blue. YMMV, definitely. Had a couple of boxes of blanks that totally oxidized - they looked cloudy like they got wet, but the boxes were fine, so I know they didn’t get flooded.

Thanks for the link - makes for a great trip down memory lane on a slow workday. :slight_smile:

oh, thanks for the offer though :slight_smile:
memory lane, i guess thats what a lot of data is all about :slight_smile:

(btw that dvddisaster tool eec isnt to add a 2nd session to an existing cd, but rather to save off eec data to somewhere else.
eg an iso around 500meg generated about 60meg of eec data)

Oooh… Well hmm… If it’s saving ECC elsewhere, I guess it’s just a matter of maintaining that database.

Very RAID 3 ish. :slight_smile:

Whatever way you choose, definitely keep us updated on your status - this is a problem others will run into soon as well.

…and I contribute a link back: Dungeon Master Encyclopaedia

thanks for the link :slight_smile:

sure will let you know how things go on with time.
(so far i found a bunch of Verbatim Metal Azo’s, which say 100 years life so will put those to the test) :slight_smile:

dvddisaster i think only works on data discs which are single session, but will save the eec files to drobo - and that should be safe enough.

(i have another thread about 2nd volumes which will be updating soon as i insert the 4th drive into my v1 drobo)

Hi guys, heres some info so far:

I tried about 15 cds so far, ranging from creation date 1997-2010
bd=burn date (where i could determine the date the cd was first burned etc)

apart from some dodgy CDWR’s (maybe failed burns or something), which had unreadable sectors - which ive since done a full erase on them and found out that CDRW is for much more tmp use instead of actual storage (eg burn and take somewhere and use quicker)

all the other cds tested so far were 100% readable
eg all green acording to DvDisaster in terms of all the sectors and blocks.

Tested samples were these so far:
Infinity-pro-dark_grey_dye
(Burn Date =est. 2001)

verbatim-52x-700mb-silverbottomdye
(bd-aug-2010)

TDK-bluetop-Reflex-deep_blue_dye-650mb
(bd-nov-1998)

nobrand-deep_blue_dye-silvertop-seethru
(bd-nov-1998)

nobrand-deep_blue_dye-silvertop-seethru
(bd-jan-1999)

and Oldest so far (and most expensive cdr LOL £3.50 - £5 for a single back then. (burned probably with a Ricoh model)

SONY-goldtop-pro_use-printable-deep_GREEN_dye-650mb-cdq-74bp
(bd-jan-1998) made_in_japan

SONY-goldtop-pro_use-printable-deep_GREEN_dye-650mb-cdq-74bp
(bd-feb-1998) made_in_japan

TDK-whitetop-Reflex-deep_blue_dye-650mb
(bd-may-1998)

TDK-greentop-Reflex-deep_blue_dye-650mb
(bd-jun-1998)

DATASAFE-80mins-whitetop-light_green_dye
(bd-fairly_recent-0-5y)

only the following cd so far, looks like a possible candidate of a future failing, acording to the visuals of the graph/curves, though all sectors were green.

nobrand-deep_blue_dye-silvertop-seethru
(bd-dec-1998)
all green-but-ending-fluctuates-x2 (repeated and similar results)

can i email one of you a bmp graph for us to add to this thread and i can show what i mean?
ahh found attachment feature on forum :slight_smile:

[attachment=21]

(ive got it down to about 8-10mins per cd to use DvDisaster to generate an iso and eec onto the drobo and to then verify it off the drobo)

estimate a long time to go through things LOL
but so far so good :slight_smile:

well i know what you will be doing NYE :stuck_out_tongue:

I found more info about that Sony cdr it got a good review LOL:

"
Name Sony CDQ-74BP Printable
Factory Taiyo Yuden Company Limited (Type 1)
Color Top Gold
Color Label Beige/Gold Text
Color Dye Green
Free Sectors 336000 656.2 Mb (74:40)
Total Sectors 336075 656.3 Mb (74:41)
test CD-Writer used HP4020i
OverSize 77:07
Info: Very reliable CDR
"[hr]
hopefully not docchris :slight_smile:

I found my first BAD cd :frowning:
luckily i was able to extract an iso and access most files on it except a few folders which arent that important.

that was about the 21st disc i tried… so i guess 1 failure in every 20 good cds so far isnt that bad
the bad one was this:

samsung-blue_spindle_logo - deep_blue_dye-metalic_coarse_silver_top
(bd-Oct-1998)
15 bad sectors (6 bad spiral blocks) - 304 unreadable sectors

attaching pic here for you guys just fyi (clickable for zooming to see the red blocks)[hr]
one good feature of DvDisaster is that within 3 mins of scanning (it also saves the iso as it goes along, to the drobo) :slight_smile:
then another couple of mins, say 3, to generate EEC from a good cd (also to drobo as a file)
plus 2mins to re-verify and a couple of mins overhead on my part to load in a cd, swap out a cd, and make a note that its been analyised and make a new folder etc, and thats the 8-10mins

so far 17 cd’s dumped on drobo this way (plus a few were just EEC files generated) = around 11GB which the drobo can handle no problem :slight_smile:

im sure that ISO is a compressible format, so will be trying a test using NTFS folder/file compression on selected folders to see what savings i can make, and i reckon this is the way to go.

Not bad… 2011 - year of data transmogrification! :slight_smile:

ISOs generally are only as large as the data, so they’re sort of tar-ish, if you’re familiar with *nix systems and tend to compress, unless the content is non-compressible stuff.

First burner of mine was a SCSI Ricoh MP6200S… I was probably one of the first few people in Hawaii to have one. :slight_smile:

cool, you probably had to stick a “terminator” on it :slight_smile:

ok guys, ive realised that after doing a fair few dumps in this way…
its actually better in the long run to:

-make the iso
-generate the eec data
-dump/extract the data itself on to drobo
-delete the iso

this way the data files will be in a ready-to-use state (eg no iso mounting tools to use etc)
they will be quickly searchable (like a normal computer search) (eg using that cathy tool i mentioned (much better than windows search)
and i’ll also have the eec in case i rearrange things on the drobo and need to recover a particular data disc with same exact content it had before etc.

(and ntfs compression hasnt helped much, takes a fair amount of time for hardly much saving - but i wonder what its like on a whole drobo - hmm i’ll have to post another post :slight_smile:

i’ll keep going through things and update this periodically :slight_smile:

Now that it sounds like you have a proposed flow, it’s a good time to simulate a disaster and see how the recovery holds up. :slight_smile:

ahh thats true… there was this “TAO guide to backups” or similar on a web page (not sure if one of you guys first posted it but it was about this kung fu master of backups, and his tao apprentice and how he learned (the hard way) to follow the path that leads to enlightenment :slight_smile:

first things i need to do, is to upgrade my current DR setup… at the moment i have:
a v1 Drobo (site 1)
a v2 Drobo (which the v1 is mirrored onto) (site 1)
mozy online backups (site www web) (but ONLY about 0.8% of my data so far (most is unsorted, dumps of work in progress etc uncataloged etc etc)

my next plan is to be able to have a 3rd (4-slot) drobo v1 or v2 etc, which is actually at site 2…
BUT
ive yet to find a (decent, reliable) hosting company which is willing to host such a drobo hardware. even tried Rackspace, and they refused to host anything other than their own hardware :([hr]
http://www.taobackup.com/[hr]
(maybe what we need is DRI to get into the hosting services, or DROBO users to host each other’s drobos lol :slight_smile:
could even make a TV series out of it… and call it “The Little-est Drobo” :smiley:

another bit of info to help confirm why its not so good to just save the iso…
when i looked into it, lots of filenames became truncated like the dos zbcdefgh~ names