How Drobo manages space: Is this theory correct?

Forum History General Discussion
1

I posted a (specific) question last night which was really indicative of my lack of (more general) knowledge about how the Drobo manages space. I was thinking about this more and I think I’ve come up with a theory – I’d love it if someone could confirm or deny this. (Then, my other question would be answered, too)

My theory: Every Drobo volume can have two kinds of partitions in it – for lack of better names I’ll call them “sparse-friendly” partitions and “sparse-unfriendly” ones. The Drobo understands the filesystem format of sparse-friendly partitions, and is able to store them on its physical storage in a sparse manner. So you can have a sparse-friendly partition that is much larger than the free space on your Drobo, and that works just fine.

Sparse-unfriendly partitions, by contrast, are “wired” – physical space is allocated for them immediately and stays allocated to them forever. The Drobo doesn’t necessarily understand the filesystem format.

If you remove a disk (and don’t replace it, but just wait until everything is green again) so that the amount of total space is permanently lowered, then you’d better have enough space to cover all your sparse-unfriendly partitions plus all the used space in your sparse-friendly partitions. (Or else what? I’m guessing it just means the Drobo will remain in degraded mode until you add another disk)

So the instructions floating around the net about how to create a Time Machine partition (which say there’s something magical about the “last” partition in a volume) are incorrect. It’s not the fact that it’s the last; it’s the fact that it’s sparse-friendly, whereas the first partition may or may not be.

If all of this is correct, then my one question is: What exactly determines whether a partition is sparse-friendly or sparse-unfriendly? Is it the filesystem format (e.g. is an HFS+ partition created by Disk Utility just as good as an HFS+ partition created by Drobo Dashboard)? Is it who created it (e.g. an HFS+ partition created by Drobo Dashboard will always be sparse-friendly, and an HFS+ partition created by Disk Utility will never be)?

Is there a danger that you might be able to somehow create a partition that the Drobo thinks is sparse-friendly, but really is in a format that the Drobo doesn’t quite understand (e.g. case-sensitive HFS+), and so it might incorrectly think that a certain block is free (and doesn’t need to be backed by actual storage) when it’s actually not free? Or is the Drobo pretty good about determining what’s sparse-friendly and what it should just allocate space for up-front and then leave alone?

Is there anything inherently unsafe about using sparse-unfriendly partitions? I’m trying to figure out why people are so adamant that “using Disk Utility is a bad thing”.

Or is my theory totally incorrect? :slight_smile:

Thanks!!

2

As far as i know, your theory is not correct, drobo neither understands nor supports what you refer to as “sparse unfriendly” (in fact i don’t understand what you classify as this)

What makes you think that such a thing exists?

3

and rather than sparse/non-sparse, you mean thin-provisioned versus , i guess, full-provisioned (i.e. they actually take up the the space they claim to have)

as far as i am aware all drobos only support thin provisioned volumes.

4

Hi Docchris,

Thanks very much for the response! Can you explain how things do work, though? What is the Drobo actually doing to figure out which filesystem blocks need to be backed by real storage? If it can’t figure it out (e.g. if you format a partition using an unsupported filesystem type) will it “just not work at all” or will it work, and just assume that all blocks which have ever been written to need to be backed by real storage?

What is the difference between a filesystem created by Drobo Dashboard, and one created by Disk Utility (which some people say is unsafe, but which other people recommend in a limited fashion in some cases)? What makes it dangerous, if it is in fact dangerous?

If there were a whitepaper or something on how this stuff works, I would love it if someone could point me at it. I have the feeling that I’m groping in the dark for the right questions to ask, when what I really want to say is “tell me everything!” :slight_smile:

5

Hehe, “tell me everything” kind of goes against “easy black box” of Drobo.

Likely format of an unsupported filesystem will result in one of two things:

  1. Format will fail because Drobo will return invalid results for the format’s write operation once it runs out of physical space.
    For example, Chkdsk /R (scan for bad sectors) will return a TON of bad blocks when it’s checking blocks that don’t have physical counterparts, so Drobo must be returning either a failure or some fixed pattern that doesn’t match the test pattern.

  2. Format will complete, but data saved/read may be completely unstable.

As for format/partition outside of Drobo Dashboard, it’s risky. Except for the specific procedures DRI outlines, any other format/partition outside of Drobo Dashboard is “outside of what Drobo knows” and is likely to cause huge problems down the road.

Drobo is a fixed system and behaves best when you treat it that way. Give it only what it specifies, and it will be happy. Just like your car is happy with the specified fuel and lubricant. If you make your own fuel and lubricant and feed it to your car, it might die immediately, or worse, it might work for a while then suddenly die.

Treat Drobo like a medical device. Use it within its specified parameters, don’t try to hack it. I hope people don’t hack their pacemakers (though I’m sure my Google search in the next minutes will find a few).

6

agreed with bhiga.

drobo works by querying the filesystem to find out what blocks are in use and it therefore needs to protect

i suspect that using an unsupported filesystem would cause it to fail, either at the formatting stage, or if you did manage to format it, pretty much immediately after that.

you are relying 100% on drobo to look after ALL the data - any blocks in use, drobo has to 1) protect and 2) return when asked using an unsupported filesystem and you cant rely on drobo to do either of those things! obviously this totally defeats the point of having a drobo

7

@tamino,
Drobo understands the file system data structures for HFS+, NTFS, FAT32, and EXT3. That is how it knows how much capacity is used. For other file systems, you can format and use Drobo with them, but if you do Drobo will not know how to reclaim space from files that are deleted. This means the capacity indicators (blue lights, Dashboard, etc) will behave in a funny way – Drobo will seem to get full. The OS will reclaim and reuse blocks, but the indicators will never move down from their high points. Everything is OK, except for the capacity indicators.

8

Mark,

Thanks!!! That really helps. I had my heart set on using case sensitive HFS+ for at least one of the volumes, and I totally don’t mind that it won’t be able to reclaim unused space.

I’d actually rather “live within my means” and, if I know that I have 5.4 TB usable space, only use the first two of the 2TB volumes, and leave the third empty and untouched – that is, until disks get bigger and I’m able to put enough storage in to get above the 6 TB mark. That way I know I’ll never hit a “slowdown”, too. The OS will tell me I’m out of space and that will be that. That seems safer to me, actually.

(And I don’t mind about the capacity indicators, either – I’m happy to look at the OS to find that stuff out).

You rock!!

9

Mark, if the Drobo cannot properly reclaim space from deleted files, does not that mean too that Drobo-specific fragmentation will get increasingly worse, with lower performance as a consequence, when modify/write activities are high ?

10

@geeji,

Are you considering using Drobo with a different filesystem? Which one?

There is no answer to your question other than “it depends”. Factors include how data was stored – lots of small files, or large files, the access pattern leading to the modify/write actions, etc. Some cases may cause insignificant performance hits, others huge.

11

Mark,

Are the performance/defragmentation benefits of using a supported filesystem for one partition negatively affected by using an unsupported filesystem for a different partition?

For example: Let’s say I have an unsupported partition, X. Drobo can’t see into the meaning of any of its blocks, so it just knows “virtual block 13 maps to physical block 48, virtual block 57 maps to physical block 9” or something.

Let’s also say I have a supported partition Y. Drobo is trying to reorganize things to make the physical blocks of Y’s files contiguous. In order to do that, it needs to move some of X’s blocks out of the way.

It should be able to do that just fine, right? Because as long as the OS can access X’s virtual blocks 13 and 57, it shouldn’t matter that the Drobo is actually mapping those to 102 and 496 now (because it needed to use physical blocks 48 and 9 to optimize partition Y).

I’m just hoping it actually does reorganize X’s virtual-to-physical mappings for the benefit of other filesystems that may need the physical space that X currently occupies, rather than just going “whoa, I don’t understand this X guy, I’m just going to leave his blocks completely alone. If he’s in the way, oh well, other filesystems will just have to deal”.

The reason why I’m so intent on this, and this is speaking for myself only, is… I’m planning on using one of my 2TB volumes for VMware images, so case sensitivity completely doesn’t matter there and I’d like it to be fast, so I will definitely leave that one as HFS+. But I’d really like to be able to export the other 2TB volume as a household file share. Performance would be (relatively) unimportant for that, but I’d want to use case sensitive HFS+ so that people using the file share wouldn’t be confused by the case-insensitivity.

If this is really, truly a bad idea, then I’ll skip it (or I’ll see if OS X will let me share a mounted disk image). It sounds like it’s quite possibly not bad at all, though.

12

why would people be confused by case insensitivity? if anything i’ve found people get more confused by case sensitivity

if you are manually typing in a file path, its much easier just to be able to bang it in and not worry about the case, rather than trying to remembered what is capitalised and what isnt.

or have i completely missed the point? (it woudlnt be the first time!)

13

Hi Docchris,

[quote=“Docchris, post:12, topic:1716”]
why would people be confused by case insensitivity? if anything i’ve found people get more confused by case sensitivity[/quote]

Heh – now we’re really off topic for Drobo, but… :slight_smile: I’m not sure if you’re a Mac or Windows user – if you’re on a Mac, try this (you should have a Pictures directory (with a capital P) inside your home directory, because it’s one of the ones that Apple sets up by default.

Type “ls -d pictures” (lowercase p). It shows up fine. It looks like it has a lowercase p (because that’s how you typed it, and ls has no reason to rescan the directory to find the “real” name – it just did a stat on the exact filename you gave it).

Now type “ls -d pict” and hit tab. You don’t get anything. Wait! Where did it go? It was there a minute ago!

If you had typed “Pict” before hitting tab, it would have shown up. But you had no way of knowing that, short of doing an ls on the entire directory to find out what the capitalization actually is.

If you’d been on a case sensitive filesystem, the behavior would have been more consistent – “ls -d pictures” would have said “no such file or directory”, so you’d have gone looking for it, and when you found it (with its capital P) you’d then have been aware that it had a capital P, so you wouldn’t have been trying to tab complete it with a lowercase p.

Another of my favorite examples is the build process for Python – the source distribution contains a directory called “Python” where all the interesting bits live (the Modules directory, the Objects directory, etc…) – and then when you build it, it produces an executable named “python” (lowercase p). It’s expecting that that will work fine, because it’s expecting that your filesystem is case sensitive. But with a case insensitive filesystem, it just doesn’t work, because you can’t have something named Python and something different named python at the same time.

At least I know with a case sensitive filesystem, what I see is exactly what I get.

I guess I should probably file this as a feature request, huh? Official Drobo support for case-sensitive HFS+? (ObDrobo :-))

14

sorry, windows guy here :stuck_out_tongue:

15

Ouch Mac OS behavior is confusing at best. Case matters in tab-completion but doesn’t matter elsewhere? That’s just plain mean.

Yes, feature request would be best. :slight_smile: