Drobo

New encryption method for log file

This question is directed at the forum moderators, if they are allowed to answer.

Does the company foresee opening up access to the log file in the future?

I spent several hours tonight examining the encrypted data and was able to figure out the new encoding scheme for the diagnostics file and code up a decoding engine for it. I assume the reason DRI moved from the single XOR to a larger pad was that DRI felt the log file should not be publicly readable.

Is this the long term plan, to keep the log file available only to DRI support? Or, does DRI foresee opening up access to the log outputs in the future?

-brendan

PS - note that I do not intend to release the code, since I assume the reason the developers went with a more complex encoding scheme was to keep the log files private.

According to a response I received from someone in Tech Support, DRI considers some of the information in the Diagnostics files proprietary, and therefore they have no intention of releasing them.

I don’t have a problem with that, ASSUMING:

  1. DRI improves the Drobo Dashboard to provide detailed device error statistics and failure logs. This is something the user community has been demanding for nearly two years.
  2. DRI will modify their EULA and their warranty to make it crystal clear that NO user data is ever included within the diagnostic logs sent to DRI for analysis, and people such as yourself can test and confirm that that is the case. That includes data that was recorded on the Drobo itself, as well as any data that might be lying around within the computer that is running the Drobo Dashboard. The reason I make this point is that before DRI encrypted the logs, I DID see file names and other information within a log file that I submitted, although I believe it may have been something that was picked up accidentally, within Dashboard.

If we do not see a reasonable commitment forthcoming to those two principles, I think it might be appropriate to revisit the issue of publishing the decryption code. I’m not a lawyer, but I don’t believe that either the Digital Millennium Copyright Act or the Computer Fraud and Abuse Act would apply in that case. However, you might want to review the EULA, to see what it says about reverse engineering the code, although you really aren’t reverse engineering the code in that case – only the log file.

Yes, but publishing the code I wrote would be almost pointless, as DRI would just change the XOR multibyte sequence used in v1 (or worse: actually implement a contemporary encryption library). They went with obfuscating the XOR approach after the php code for v0 of their encryption was sent out (which used to be just the single XOR byte). If they really used contemporary encryption techniques, it would be basically hopeless to try to break it. I’d prefer that not happen.

What I’d prefer to see is that they realize there’s an upside to allowing end users to understand the logged data: You end up building a supportive forum community that way.

-brendan

PS - I see SMART monitoring is enabled, but I don’t see SMART data being reported in the diagnostics results. Perhaps they are working on it … or they only report it on problem disks?

And just to clarify: from what I see in the forums the new “encryption” scheme (v1) may only be in place on the DroboPro units.

-brendan

Drobo & DroboPro can’t collect user data - they are block level devices and know nothing above that level.

  1. Thanks mark for commenting on this thread and addressing Suite B’s question. On that topic, looking through my drobopro log, I don’t see any references to any file in my filesystem.

I suppose the occasional hard crash might potentially write random memory locations to the log and those might include file data or filesystem metadata, but that’s not a very common occurrance. e.g., this excerpt is what was produced when I inserted a bad SATA disk that causes immediate shutdowns of devices it is plugged into:

*** SETTING drives power, current 0x1f, new 0xf, [4, 0x10] DM::DoHotPlugDisk (REMOVE) 2009-09-16,18:13:57: DiskManager.cpp:2854 DM: Disk 4 removed before it had become ready, or after it had been expunged. 2009-09-16,18:13:57: DM::UpdateMap GetDriveUsageBitmap 2009-09-16,18:13:57: SPMM: RemoveDisk SlotStatus[4] = 0x0 HotPlug: Msg Rcvd Adapter 0 Drive 0 Event 4 -4}}16-97:7::4?? Cvyino$fm mosk{seg}on 3=09}r9-y,1j50~73:r^ 2009-09-16,18:14:11: -------------- DELAYING BSP_EVENT_DISK_CHA24>?-09=16,=7:??:<?: 9|msuwri}wenUmme~amyv= {2vu)-zy-1~,17z}4:5w: =. >=:8?}9"4:85;x 0z852}:0 Log Generated: THU JAN 01 00:00:00 1970 -------------------- CRASH LOG FLASH FILE END ----------------------

It shows that uninitialized memory might get placed into the log under unusual circumstance, but that’s not the same as purposefully recording user data in the log, and from my perspective, is not a big deal (others may disagree).

  1. Would it be possible for you to also address the questions I put forth in the thread opener? If not, then just send them up the chain so that management knows we’d like to see the device log for the devices we own.

In any case, again, thanks a bunch. :slight_smile:

-brendan

Mark, I certainly respect you, and what you say, but I don’t understand.

If you mean that the Drobo does not examine or record the contents of individual sectors, I can certainly understand and accept that. For one thing, where would it put it all, and why bother?

However, the Drobo is OBVIOUSLY more than just a simple block level RAID system. If that were true, the Drobo would be completely insensitive to whether the drives were formatted for HFS, NTFS, EXTx, ZFS, or whatever. But it isn’t. For one thing, it would be completely clueless about the total amount of disk space used, or when files are deleted.

So it simply MUST examine the file structure, to at least some level of depth, and I assume, at least, that it has access to file names, and assuredly Volume names.

Now, I not in the habit of creating files like TerroristBombPlans.doc, or XXX3wayPorn.m4v, MyRealIncome-Don’tShowIRS.xls, or LoveLetterFromMyHoney.msg. But if I were, or even if I merely referred to some company proprietary information in a file name, I would be rather concerned if those file names showed up in a Diagnostic file sent to DRI!

So maybe in fact Drobo DOESN’T collect or report such information, but how can we know for sure? If I were involved in a nasty divorce suit, and I learned that my spouse had recently sent in a Diagnostic file, I would be tempted to obtain a subpoena, and go fishing, just in case.

(You don’t have to be crazy to be paranoid! :slight_smile:

bhoar - I don’t foresee DRI opening up the log file. We are rethinking how drive status information is reported to users.

SuiteB - Drobo doesn’t care what file system you use. Its a block level device. There are four file systems (EXT3, FAT32, HFS+, NTFS) for which Drobo can read the filesystem bitmap, i.e. which blocks are in use. This enables Drobo to know how full it is and display “feed me a drive” LED colors, or the blue utilization lights. There is no reason for Drobo to know how to read/write file system data – i.e. your personal information – so it doesn’t.

[quote=“Mark_F., post:8, topic:504”]
bhoar - I don’t foresee DRI opening up the log file. We are rethinking how drive status information is reported to users.[/quote]

  1. Fair enough, I guess. Perhaps regular reviews to allow moving more of the log data into the clear portion of the diagnostic file over time, as it’s potential market compete potential is reduced (for whatever reason), could be considered.

  2. I’d definitely like to see a drive reliability rating based on the drive behavior and the SMART data. At a minimum, the smart data should be available via the dashboard.

Thanks for listening.

-brendan

[quote=‘Mark F.’ pid=‘2204’ dateline=‘1253729680’
SuiteB - Drobo doesn’t care what file system you use. Its a block level device. There are four file systems (EXT3, FAT32, HFS+, NTFS) for which Drobo can read the filesystem bitmap, i.e. which blocks are in use. This enables Drobo to know how full it is and display “feed me a drive” LED colors, or the blue utilization lights. There is no reason for Drobo to know how to read/write file system data – i.e. your personal information – so it doesn’t.
[/quote]

OK. I agree about the CONTENTS of the file, but maybe there is one more thing I don’t understand about file systems, and in particular the file name (and perhaps other metadata.)

I’ve never heard of a filesystem bitmap being created by the OS, at least with NTFS or FAT, with which I a little familiar. On the other hand, I can easily understand how Drobo itself might see a command to write to a new sector, and put a mark on the wall that says that sector is now in use, and so the blues could keep track.

But how then would Drobo know that a file had been deleted? In that case it normally isn’t overwritten, it is merely deleted from the File Allocation Table, and the OS keeps track of the unused space. Perhaps I’m wrong, and the unused sectors are chained together is a free-space list without file names? But I thought that normally they merely put a mark on that file’s name to say that it the space previously used was now free, and until that is actually used, the file could still be recovered by name?

NTFS definitely has a volume bitmap - you’ll see a message on occasion about errors in the “volume bitmap” being corrected if you run CHKDSK after a power failure, etc.

[quote=“Suite_B, post:10, topic:504”]OK. I agree about the CONTENTS of the file, but maybe there is one more thing I don’t understand about file systems, and in particular the file name (and perhaps other metadata.)

I’ve never heard of a filesystem bitmap being created by the OS, at least with NTFS or FAT, with which I a little familiar. On the other hand, I can easily understand how Drobo itself might see a command to write to a new sector, and put a mark on the wall that says that sector is now in use, and so the blues could keep track.

But how then would Drobo know that a file had been deleted? In that case it normally isn’t overwritten, it is merely deleted from the File Allocation Table, and the OS keeps track of the unused space. Perhaps I’m wrong, and the unused sectors are chained together is a free-space list without file names? But I thought that normally they merely put a mark on that file’s name to say that it the space previously used was now free, and until that is actually used, the file could still be recovered by name?
[/quote]

Simplistically, when the OS deletes a file, in addition to updating the directory tree, it also clears the bits in the volume bitmap to show that clusters used by the deleted file the are no longer in use. The volume bitmap is stored separately from the file/directory/properties metadata. Programs that look directly at thedisk (or the storage monitoring routines on the drobo) don’t have to have any exposure to the filesystem metadata (file/directory names and properties) to see those 1 bits turned to 0 bits and know that certain clusters are now free. That is, once those bits are 0, then the (virtualized) clusters they map too are free for reuse in later allocations.

The important concept to get here is that the space usage table is kept separately from the other filesystem metadata.

-brendan

Well, I guess you can teach old dogs a new trick! Thanks for the info.

If it’s true that the bitmap is being used for the free space indication only, how the Drobo manages physical sectors reusage? For example, I have a 16TB volume but only 1TB of physical disk space. First I write 800GB of data, then delete 400GB. Then I write again some 400GB of another data, but this time, for whatever reason, the OS picks completely different area of the virtual volume, let’s say the clusters located at the very end of the virtual 16TB volume. It won’t be denied and the transfer will succeed, right? If so, the importance of the volume bitmap is way more serious for Drobo than for merely displaying the space used, right? And what happens when the NTFS bitmap gets corrupted in the way it’s indicating some of the used space as free (which indeed happens from time to time AFAIK) and the Drobo reallocates it for using as another range of virtual volume’s clusters? I can imagine things could get quite messy if that happens… Even worse, that kind of condition could go unnoticed by the filesystem checking tools as from the filesystem integrity point of view, everything would still be fine, just some range of the virtual volume’s cluster would point to some random trash. I could even imagine a scenario of propagating such data corruption to all the backup locations in a good will, because it’s simply impossible to check each and every file if it actually still contains the data it is supposed to. Now THAT is scary… Now I cannot stop thinking about some $BITMAP error I came across some time ago… Or maybe my understanding is wrong and my hypothetical horror story completely unfounded? Honestly, I’d rather be wrong, otherwise I’ll consider checksumming each and every piece of data I put on Drobo…

My gut feeling tells me that Drobo works with the supported filesystem of its volume.

IOW, Drobo doesn’t have its own volume bitmap - it uses the one for the filesystem. Internally it may remap physical clusters as necessary to stay within the bounds of its physical storage, but since the host system can’t bypass Drobo’s translation, no matter what the host does (beyond trying to fill beyond the amount of available storage), Drobo will keep it “in line.”

I’ve never seen the volume bitmap mark used space as free (which would essentially mean files are “lost”) but I have seen it mark free space as used. In the case of having free space marked used, Drobo would simply think it’s “more full” than it really is, and it might issue the “Hey, I’m getting full” warning earlier than needed.
Once the volume bitmap is repaired and the free space marked as free again, then the free space usage will be corrected, Drobo will know there’s more free space, and it’ll then be “less full” again.

Or I guess another way to look at it would be like Drobo is a managed (non-accessible, ie, you don’t have “your own” shed/locker) public storage facility and you’re the OS.

You take your goods there and ask the agent there to store it.
They generate a manifest, number each of your boxes and that becomes the manifest (volume bitmap) and that manifest manages how many boxes you have stored and of what sizes (volume bitmap), because you only pay for a certain amount of storage.

Internally, you have your own sheet of what’s in each box and their groupings (filenames and content) - kitchenware #2 of 5, bedding #4 of 7, toys, etc. But you don’t share that with the storage facility because they don’t need to know.

The storage facility, in an effort to maximize space efficiency, will split up your boxes and put each one where it fits best. Box 1 might end up in warehouse 8, aisle 4, shelf 1 while Box 2 might end up in warehouse 13, aisle 2, shelf 4. The storage facility manages the physical locations of your boxes (physical layout), and you have no idea of where your stuff is physically, other than it’s with the storage facility.

Since the storage facility might not actually have enough physical space to hold the entire storage space that you have paid for (thin provisioning), they keep an eye on the amount of used space compared to the amount of actual space they have (keeping an eye on the volume bitmap). If they start to run out of space, they raise a flag that they’ll have to build/acquire more warehouse space (“hey I’m getting full” indication from Drobo).

When you come to retrieve a box, you just tell them which box from the manifest you want, they check their internal list to determine where to physically grab it from, then they deliver it to you and mark it off the manifest as free space.

In the event that somehow the manifest becomes out of sync with what the facility actually has (maybe you wrote down a box and just before giving it to them, decided not to and didn’t erase the box from the manifest), then you read off each of the boxes and the facility does an internal physical check to see if the box is there (CHKDSK/fsck).

If they find there’s a box on the manifest that isn’t actually in the warehouse, then they fix (repair free space marked used) the manifest. (We’ll assume for now that theft and loss don’t happen in this universe)

Since the storage facility doesn’t let you directly rummage through the warehouse and everything you do is through the manifest and facility agent.

Would be great to get a confirm/deny from DRI.

Brandon

First off - Hi Mark! nice to see you back on the forums again :slight_smile:

Second - while im am happy to accept that some parts of the log DRI consider to be proprietary, the parts of which which relate the the health of MY drives, holding MY data, i would consider that i have a right, and probably even a need to see (plus im naturally a curious person and will almost always take things apart just to see what’s inside).

perhaps going forward even have it generate a pair of log files? a DRI only one, and then a more basic (and possibly marginally easier to understand) “user” one with Drive health/drobo health (does it log temperatures?)/drive status log.

My 2 cents

Debate about log files is misguided, Dashboard needs to provide more status information. Tell us, Drobo, what you know about the drives in the system.

I imagine that part of the reason for encryption is to stop lots of panicked calls from improper interpretation of what is reported in the log.

but instead they get lots of people asking them to interpret the logs for them.

im not sure thats better

please?.. :slight_smile:

I definitely like the idea of a “filtered” log that is user-accessible.

The “filtered” log would obviously exclude anything that puts DRI-proprietary stuff at risk, but definitely should include information about the drives. Perhaps a default “threshold level” could be set so as not to panic curious newbies.

Now, if DRI could establish an agreed-upon set of filter rules, then maybe bhoar could implement them and release binaries or even a web submit-and-return?

Just a thought…