Some folders (not all) and Files disappear in mergerFS after removing the 2 empty drives of the 3 drives in the pool but remain on the drive.

[ShadowWizard11]

Describe the bug

As a result of a hardware failure, while I wait for parts, I cannot hook up all my drives to the system. Since I had 2 empty drives in a mergerFS, I chose to remove them. MergerFS will show the folders on the remaining drive, but not the files. Files have been confirmed to exist, and work.

To Reproduce

Can't really say. I am using Open Media Vault, and they suggested I come here.
Have a hardware failure, and when you turn the system back on, only have 1 of the 3 mergerFS drives connected maybe?

Expected behavior

MergerFS should show all files within the folder of the remaining drives.

System information:

• OS, kernel version: uname -a

• Linux shadownas 5.4.203-1-pve overflow risk in ioctl.cpp #1 SMP PVE 5.4.203-1 (Fri, 26 Aug 2022 14:43:35 +0200) x86_64 GNU/Linux (Newer kernels seem to have issues with my system. Kernel upgrade is not an option)

• mergerfs version: mergerfs -V
  mergerfs version: 2.35.1 (This is the version offered with Open Media Vault. They offer no updates)

• mergerfs settings

• List of drives, filesystems, & sizes:

  • df -h
    Filesystem Size Used Avail Use% Mounted on
    udev 2.9G 0 2.9G 0% /dev
    tmpfs 594M 5.4M 588M 1% /run
    /dev/sda1 109G 29G 75G 28% /
    tmpfs 2.9G 84K 2.9G 1% /dev/shm
    tmpfs 5.0M 4.0K 5.0M 1% /run/lock
    /dev/sdi1 5.5T 1.1T 4.4T 20% /srv/dev-disk-by-uuid-0f51da5c-e4ae-45cb-aebb-30c6275a5451
    tmpfs 2.9G 268K 2.9G 1% /tmp
    /dev/sdg1 13T 1.3T 12T 10% /srv/dev-disk-by-uuid-ac7b71aa-0987-4ba0-8493-ea1943c17a71
    mainpool 5.8T 361G 5.5T 7% /mainpool
    mainpool/Config 5.5T 26G 5.5T 1% /mainpool/Config
    mainpool/homebkup 5.8T 263G 5.5T 5% /mainpool/homebkup
    mainpool/storebkup 5.8T 301G 5.5T 6% /mainpool/storebkup
    mainpool/media 6.6T 1.1T 5.5T 17% /mainpool/media
    mainpool/Main 8.8T 3.3T 5.5T 38% /mainpool/Main
    mainpool/dnd 5.5T 24G 5.5T 1% /mainpool/dnd
    mainpool/dndbooks 8.7T 3.2T 5.5T 37% /mainpool/dndbooks
    mainpool/Torrents 8.3T 2.8T 5.5T 34% /mainpool/Torrents
    mainpool/test 5.5T 6.7G 5.5T 1% /mainpool/test
    mainpool/OMVBackup 5.6T 150G 5.5T 3% /mainpool/OMVBackup
    mainpool/Laser 5.5T 15M 5.5T 1% /mainpool/Laser
    mainpool/Modules 5.5T 256K 5.5T 1% /mainpool/dndbooks/DnD-3.5/Dungeon Master Tools/Adventures
    (The only drive remaining that is part of the pool is sdg1)
  • lsblk -f
    NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINT
    sda
    ├─sda1 ext4 1.0 622fccb7-e715-4103-9463-96cf97406843 74.3G 26% /
    ├─sda2
    └─sda5 swap 1 627364b9-f1bd-468d-a83b-8f19656d9cb3 [SWAP]
    sdb
    ├─sdb1 zfs_member 5000 mainpool 2221449291355633664
    └─sdb9
    sdc
    ├─sdc1 zfs_member 5000 mainpool 2221449291355633664
    └─sdc9
    sdd
    ├─sdd1 zfs_member 5000 mainpool 2221449291355633664
    └─sdd9
    sde
    ├─sde1 zfs_member 5000 mainpool 2221449291355633664
    └─sde9
    sdf
    ├─sdf1 zfs_member 5000 mainpool 2221449291355633664
    └─sdf9
    sdg
    └─sdg1 ext4 1.0 ac7b71aa-0987-4ba0-8493-ea1943c17a71 11.4T 10% /srv/dev-disk-by-uuid-ac7b71aa-0987-4ba0-8493-ea1943c17a71
    sdh
    ├─sdh1 zfs_member 5000 mainpool 2221449291355633664
    └─sdh9
    sdi
    └─sdi1 ext4 1.0 sixtb 0f51da5c-e4ae-45cb-aebb-30c6275a5451 4.3T 20% /srv/dev-disk-by-uuid-0f51da5c-e4ae-45cb-aebb-30c6275a5451

• A strace of the application having a problem:

  • strace -fvTtt -s 256 -o /tmp/app.strace.txt <cmd>
  • strace -fvTtt -s 256 -o /tmp/app.strace.txt -p <appPID>

• strace of mergerfs while app tried to do it's thing:

  • strace -fvTtt -s 256 -p <mergerfsPID> -o /tmp/mergerfs.strace.txt
    Not sure this applies, as even logged in as root heading to /srv/mergerfs and down to the appropriate directory shows no files in that directory.
    Additional context

In case I wasn't clear (As I know I can be some times) I will try and clarify the details here.
Merger FS is set up to use 3 disks:

/srv/dev-disk-by-uuid-ac7b71aa-0987-4ba0-8493-ea1943c17a71
/srv/dev-disk-by-uuid-411d2061-8470-4fde-b61f-1323387c60f0
/srv/dev-disk-by-uuid-a3797d83-3c8d-4840-981a-f67ba1de603a

Only the 1st of those disks is in the system.
I log in as root.
When I navigate to /srv/dev-disk-by-uuid-ac7b71aa-0987-4ba0-8493-ea1943c17a71/firstexp/media/TV and do an LS a large list of directories shows up.
I am able to navigate to /srv/mergerfs/firstexp/firstexp/media/TV (No idea why firstexp is doubled, however there are no other directory options after mergerFS to choose, except in the media folder. There is a TV and a Movies in the media folder. Both are empty. TV is just used in this example)
When I do a ls, I am shown nothing. When I do an ls -ah I am shown ". .."
mergerFS seems unable to read anything below the TV and Movies folders in meda.

[trapexit]

mergerfs version: 2.35.1 (This is the version offered with Open Media Vault. They offer no updates)

Please use the latest release as requested in the docs. I don't control Debian (which OMV is based). I can't force them to upgrade. But that version is years old. I provide easy to install deb packages for all releases I make. Please start there and as suggested in the docs confirm you have no permission issues and simply the setup with a singular branch.

[ShadowWizard11]

The issue is with updating is I don't know how integrated mergerFS is into the UI or anything. MergerFS was set up with a GUI in OMV, and without the skillset to go back to the version I have now, I can't risk the upgrade. You seem to be familiar, at least a bit with OMV, so you know this is my file server, and you can likely tell by all the drives, its pretty big (by home standards) so risking and upgrade of merferFS without knowing how it will play with OMV and without the skillset to roll back I am sure you will agree isn't a responsible option on my end. I would still like to resolve this in a way that doesn't risk my file server, but if you are unable/unwilling, I can very much understand and respect that.

[trapexit]

I can't invest serious effort into debugging software that is year+ old. Even if I did investigate and found something you're saying you won't upgrade so I can't do anything about it.

You can ask on the OMV forums about upgrades but it's literally just downloading the package and sudo dpkg -i package.deb and if it didn't work you just uninstall it and install the original. mergerfs has no real dependencies. People upgrade all the time and I've never heard of anyone having issues. If you still don't want to do that then follow the debugging tips I've provide in the docs. mergerfs is a proxy. It reports what it finds. If nothing gets returned either the perms are messed up or nothing is there or you've enabled some caching and changed things out of band.

[ShadowWizard11]

You are right, in both accounts. I was kind of thinking it has to be something ducking op on my end, but I do agree that debugging an old version isn't reasonable. And you are right, if things go amiss, I can always remove it, and install the older one. I only have the one "set" of mergerfs drives, so it won't be a big deal to "rebuild" them if I have to.
That being said, I again lack a skillset. To translate this:
wget https://github.com/trapexit/mergerfs/releases/download//mergerfs_.debian-.deb
dpkg -i mergerfs.debian-_.deb

Into the actual command. Thought I may be able to navigate through it via a web browser, but unfortunatly I can't seem to go to https://github.com/trapexit/mergerfs/releases/download/ to be able to maybe fill in the blanks. Is there a "I don't really know how github works" tutorial/directions on installing it?

[ShadowWizard11]

Well, I think I found the issue.
I had a thought. If I told any program, mergerfs, or anything for that matter, to do something with something that didn't exist (start mergerfs with these 3 drives, of which 2 don't exist, or load windows from d: drive, when D drive doesn't exist) it would error out. And OMV maintained the list of disks that were missing. Or maybe it was OMV that was flipping out because those didks weren't there.
So I can't confirm 100% that was the issue, but it seems as through removing the disks that no longer exist from the config in OMV worked. In a few days when the disks are added back to the server, I will just add them back to mergerFS
Thought I would post the solution here in case anyone else has the same issue. And at the same time thank you for taking the time to reply, even though you weren't able to help.

[trapexit]

Is there a "I don't really know how github works" tutorial/directions on installing it?

It's nothing to do with github. I point to the releases page in the install section you reference and you just need to download the software. And naturally you have to know the OS you're using. I could put some instructions on finding out those details I suppose.

mergerfs doesn't care if branches disappear. If something doesn't work it ignores it. If you have 3 branches in the config and two don't have anything... it doesn't care. An empty path is just an empty path.

Also... there is nothing to "rebuild". mergerfs is just a proxy. It doesn't touch your setup. That's the point of the product. https://github.com/trapexit/mergerfs/tree/2.40.2?tab=readme-ov-file#can-mergerfs-be-used-with-filesystems-which-already-have-data--are-in-use

You can put together as many mergerfs pools with any combination of paths you want.

That said... I can't speak for OMV's webui. I am not the author or maintainer of it.