A Controller like this in Drobo?


its a long article but if you are interested in storage its fascinating. a controller which does de-dupe and loseless compression on the fly to improve speed and reduce wear on SSDs

im sure DRI could pilfer a few of those concepts (with a powerful enough CPU)

Its still be interested in getting one of those SSDs to replace my aging intel one

@Docchris, lossless compressions seems like a waste of time. Its media files that take up the space and require a Drobo, and their formats are already compressed. In an office environment, Drobo costs so much less than traditional storage that the economic need for de-dupe is reduced. Right now Drobo is held back by its processor, heaping on more tasks doesn’t seem like the way to go.

well a lossless dedupe across various media files could possibly save quite a bit of space? im sure there must be enough repeating patterns across my 9TB of media!

and i was thinking more using whatever processor they have as a co-processor :0

Yes possibly depending on your data.
Current solutions usually use a hash function to match the data you want to write to any existing data on the disk.
This has the drawbacks of requiring CPU power to calculate and the possibility for a hash collision. Not sure how a thin provisioned Drobo would do with corrupted data due to a hash collision…

If you are really interested in deduplication check out http://blogs.sun.com/bonwick/entry/zfs_dedup
ZFS Dedeuplication will be available in OpenSolaris soon (if its not already) and hopefully in an upcoming Solaris 10 release.

im fairly impressed

however of course i have no idea if i have much block level duplication or not… i know i have zero file level duplication so that would be useless to me.

8.9 out of my 9TB is heavily compressed video - a Mix of AVC, MPEG 2 & MPEG4 (and of course accompanying AC3/DTS/DTS-HD tracks - which consume a lot of space in their own right!)

my knowledge of this is hovering around the zero mark, but i understand that they are all (obviously) VERY lossy codecs, but would this imply that they are essentially breaking things down to a set of standardised chunks - many of which are possibly identical across my media collection… ?

I’ be very interested to hear the thoughts of anyone with any knowledge of video compression? :slight_smile:

of course even if the answer were yes - i have no way to implement it, its just curiosity :slight_smile:

ZFS is dead, haven’t you heard? For almost 3 yrs on this forum, and its predecessor, folks have dreamed about how good it will be when ZFS arrives. Keep on waiting.