Drobo

NFS (unfs3) and Duplicate Filenames

To make the following clearer: I’m on linux for desktops / servers, Ubuntu and Debian with Kernels 2.6.26, 2.6.32 and 2.6.36 (IIRC)

I moved my personal /home directory from a VM machine to my new drobo and have had nothing but problems since.

I know, unfs3 doesn’t support locking so I had to get around that with separate locking methods, and the nolock parameter on my /etc/fstab.

The problem is, when I get to an unknown number of files, any ‘mv’ that happens causes duplicate file names. It doesn’t matter which client I connect with so I don’t believe it’s a client kernel issue, I do not see the duplicate file names when I ssh into the drobo so I don’t think it’s a drobo file system issue, and finally in my tests with unfs3 server NOT on the drobo, I can’t get the same issue to trigger.

This causes a major pain for my mail services since once it see’s duplicate file names, it tries to resync its internal database… of course, since this is a file layer issue, it always see’s this duplicate file name and causes the imap process to hang while constantly trying to update it’s database.

You can see additional details on the bug report I started to work on via debian here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=599823

Any suggestions welcome - or hints that Kernel NFS is coming :wink:

Sigh - I’m seeing other ppl with uNFS issues, and wanted to make sure this is seen… I hope to have an answer some day…

I should also note that now this duplicate file problem is happening all over the system, it’s causing my home drives to become duplicated, causing my desktops to constantly access the drobo and causing the powersupply to overheat.[hr]
Just a quick note to say thanks Jennifer.


I made a quick call to Support knowing full well that they couldn’t do much due to the fact this is communty supported software - however, she’ll have the engineers look unto NFS on the drobo - and maybe we’ll even get real Kernel Based NFS on an upcoming release…

She also showed me how to force a firmware reload - we’ll see if that helps at all as well.

After the forced firmware update - I was able to get the issue to happen again. For those that want to test this:

Script to make a bunch of files:

for i in {1..2000} do echo This is a test file > $i.txt done

And the script to test the issue:

for i in {1..2000} do mv $i.txt $i.test.txt done

Howto:

From a NFS client machine - run the “make” script, then the “test” script - over time an ls will get slower and your start to see duplicate filenames. The only way to clear the duplicate filenames is to unmount all the shares and unload the NFS modules from your unix system.

[quote=“coolacid, post:3, topic:1917”]The only way to clear the duplicate filenames is to unmount all the shares and unload the NFS modules from your unix system.
[/quote]

Seems you have a caching problem on the client-side. Have you tried to mount the NFS share without (or with minimum) client-side caching?

I would normally agree with you - and originally I did since I opened a debian ticket for that exact issue, however, since this happens on multiple computers with different kernel levels, and it only happens with the drobo as the NFS server. I even went so far as to install unfs3 on a desktop and tried to trigger the issue with no success. (See original post)

Aside from that when looking at the PCAPs from the drobo’s NFS connection, I can clearly see the drobo sending filenames multiple times.

Sigh…

Got an email today from someone that found my debian bug tracker ticket - turns out, he has the same problem, just without a drobo. So, for the 3rd time on this issue, i’m back to square one. But at least I have chicken*… no, wait, another person that can confirm my findings :wink:

Well, it could still be a problem with uNFS, since I’m not quite sure how they implement file locking.

Traditional (kernel) NFS relies on the kernel’s own file locking capabilities, which means that it is enforced system-wide. uNFS could only possibly enforce file-locking through the NFS interface to NFS clients.

Are you by any chance accessing these shares through different protocols? It could really be that one machine using SMB or AFP is just screwing over uNFS.

Hi Ricardo;

Unfortunately, The only time locking is supported is with the kernel NFS. All the others do not support locking (Which is another beef for me, since it broke Firefox and other Mozilla apps, and my email :slight_smile: )

I run a full Linux house - I have two windows boxes, one is purpose built to remote desktop into work, and one laptop that is really used for work related. Neither of which connect to my drobo.

Hi

I reproduced the issue with Ubuntu 10.10 with 2.6.35 kernel.

I had some duplicate files. But after an unmount/remount the duplicated files were gone.

Yeah, I thought this would be the case. This probably means that uNFS has a bug with the management of file handles (if the client-side caching is not messed up).

Having recently to implement NFS myself, I can relate with the uNFS developers. The spec is awfully sparse w.r.t file handles and there are a lot of cases where I would expect that a userland NFS server would have trouble (especially on the cases where a file is not modified, but instead deleted and replaced by a new version).

Maybe trying to mount the exported folder with “noac” would solve the problem? It’ll give you crap performance, but if even this does not work you are probably out of luck.

Hi

Strange… now I’ve repeated the test multiple times (even with more file created and moved) but I wasn’t able to reproduce the issue.
The first time i run it… I stopped the make script and noticed the duplicated files… when the make script and the test script finish their task I don’t see duplicated files anymore.

Anyway it’s not an big problem for me. I’m not creating or moving lots of files on the drobo.

Cheers

I’ve been able to reproduce the issue repeatably. However, new development on the debain bug suggests that others are able to reproduce it, but not a repeatably as I am. I start to wonder if there is two issues, one server side and one client side. Both of which cause the other to be more repeatable.

I’ve now started getting the issue accross my entire home drive, so I have to reboot clients every other day or applications slow to a crawl (constantly dealing with “new” files).