Constant EXT4-fs errors in /var/log/messages - possible issues?

Network Attached Products Drobo 5N2
#1

Hi,
I’m getting the same error in /var/log/messages on my Drobo 5N NAS every few seconds:

[2022-04-03,09:01:30.671774941] Drobo5N user.crit kernel: [13620381.053960] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:01:35.653002679] Drobo5N user.crit kernel: [13620386.044715] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:01:40.702848565] Drobo5N user.crit kernel: [13620391.085251] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:01:45.710509887] Drobo5N user.crit kernel: [13620396.092925] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:01:50.682972035] Drobo5N user.crit kernel: [13620401.075060] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:01:55.677664000] Drobo5N user.crit kernel: [13620406.069292] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:02:00.664414103] Drobo5N user.crit kernel: [13620411.056758] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)
[2022-04-03,09:02:05.661701260] Drobo5N user.crit kernel: [13620416.054151] EXT4-fs error (device sda1): ext4_ext_check_inode:437: inode #29769993: comm rslsync: bad header/extent: invalid magic - magic 2f6e, entries 30317, max 11879(0), depth 26736(26736)

The errors are obviously triggered by Resilio Sync which I’m using. I haven’t found any obvious issues so far and Drobo says my drives are okay.

I also get other similar errors but less frequent:

[2022-04-03,08:58:02.429491969] Drobo5N user.crit kernel: [13620172.801622] EXT4-fs error (device sda1): htree_dirblock_to_tree:587: inode #29769740: block 1905263206: comm rslsync: bad entry in directory: rec_len % 4 != 0 - offset=0(114688), inode=1953261926, rec_len=29285, name_len=
[2022-04-03,08:58:02.547030172] Drobo5N user.crit kernel: [13620172.929835] EXT4-fs error (device sda1): ext4_lookup:1047: inode #29769740: comm rslsync: deleted inode referenced: 29770296
...
[2022-04-03,08:59:28.828723463] Drobo5N user.crit kernel: [13620259.205619] EXT4-fs error (device sda1): add_dirent_to_buf:1273: inode #19316742: block 1236271653: comm logrotate: bad entry in directory: rec_len % 4 != 0 - offset=0(0), inode=4294967295, rec_len=65535, name_len=255

I’m not very familiar with Linux. Can someone explain what’s going on there? EXT4-fs error sounds worrying. Are there issues in the virtual or physical filesystem?

Yes, I do have Backups…

Repair Filesystem: e2fsck reports errors
#2

Okay, wow… e2fsck -n /dev/sda1 ran over night and generated 694.749 lines of output.
19.174 files have x multiply-claimed block(s), shared with y file(s). Some entries are pretty ridiculous

File ??? (inode #29770006, mod time Tue Dec 28 05:48:37 1993) 
  has 125802 multiply-claimed block(s), shared with 16399 file(s):
	<filesystem metadata>
...

I guess I now need to find out what that actually means for my backups, how I can validate my backups and how to proceed with the Drobo 5N in question…

How is it even possible that the Drobo filesystem has such tremendous amounts of errors and it still runs fine like everything is okay??

#3

What’s interesting though, so far I haven’t found a single file that’s actually corrupt.
I now extracted a bunch of affected RAR/ZIP files (as they have internal CRC checksums) and they are all fine. I even found and iso image with known sha256 and it’s also okay.
How is this possible?

I’m sure all the errors e2fsck found aren’t supposed to be there?

#4

I don’t know the cause, could be almost any of the random things that sometimes happen to (any) computer filesystem…

The thing to do however, is park your data elsewhere, do a factory reset so it reformats, & then put your data back.

#5

So, I had a call with Drobo Support today and they basically told me the same.
They confirmed my filesystem is corrupted and I need to do a full reset. I didn’t really get an explanation though why e2fsck shows me more than half a million errors, while in practice I hardly notice any issues and so far the files seem to be okay. Of course it’s hard to check several TB of data in my case and for most I don’t really have checksums to start with.
They said to me the corruption is only minor and would not really affect my data as long as I’m able to access the shares. I’m not really convinced this is the case, but I can’t really prove it wrong. So, I just hope most data is fine.

#6

It really only takes a power outage or other unclean shutdown to cause this sort of thing it seems to be the commonest cause, & there are plenty of questions on places like linxquestions or stack-exchange by people using Linux as a desktop or server OS, the fix is usually reformat & restore from backup.

I would be very uncomfortable about leaving it. I’d also look at running the Drobo on a UPS in future. I think the reset / reformat needs doing though.

#7

I would be very uncomfortable about leaving it.

Me too. I’ve now put a complete set of new drives in and am currently copying data back from ma backup.
The biggest issue is that the filesystem corruption occurred two years ago (Drobo support was able to tell me that after looking at support data) but wasn’t indicated in any way. That really impacts my confidence in the system in general.