Operation not permitted when mounting as non-root

[doak]

Describe the bug

Mounting as non-root fails with fusermount: mount failed: Operation not permitted.

This fails for all versions since (including) 2.28.3-28-g5ce428c up to latest master. Especially it also fails for v2.30.0, which – according to #799 – contains a fix for some imho very similar issue.

To Reproduce

Steps to reproduce the behaviour:

mkdir branch
mkdir merged
mergerfs branch merged

Expected behaviour

Although executing as non-root is not fully supported, according to the docs mounting as non-root should work as long as there are no issues with permissions.

System information

• OS, kernel version: Linux host 6.5.0-5-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.5.13-1 (2023-11-29) x86_64 GNU/Linux
• mergerfs version: v2.38.1-1-gc1c2f07
• mergerfs settings: None I am aware of.
• List of drives, filesystems, & sizes: n.a.
• A strace of the mounting: strace.txt

[trapexit]

It works on my systems fine.

[trapexit]

46212 02:07:12.744338 mount("/tmp/mergerfs/branch", ".", "fuse", MS_NOSUID|MS_NODEV, "default_permissions,fd=3,rootmode=40000,user_id=1000,group_id=1000") = -1 EPERM (Operation not permitted) <0.000009>

From the strace. The kernel, not mergerfs, is giving you perm denied. It is clearly something about your system. Are you trying to mount this in a container? On a system without fusermount having setuid bits?

[doak]

I've found the reason (not the cause), but still don't understand it.
Anyway, you are right: it works (after I uninstalled the Distribution's version).

If you are interested in the root cause why it fails on my system, I am happy to dig deeper. Just let me know.

Some more details (in case you are interested):
Although (initially?) the expected, self built mergerfs binary get executed, it somehow "collides" with the version installed by the distribution (Debian testing). $PATH is set correctly afaik. But there is also a mergerfs-fusermount and mount.mergerfs binary installed by the package manager:

doak@flatbox:~$ which mergerfs
/home/doak/daten/software/linux/mergerfs/build/mergerfs
doak@flatbox:~$ which mergerfs-fusermount 
/usr/bin/mergerfs-fusermount

It works when mergerfs is not installed by Debian:

doak@flatbox:/tmp$ ~/daten/software/linux/mergerfs/build/mergerfs branch merged
warning: mergerfs is not running as root and may not work correctly
fusermount: mount failed: Operation not permitted
doak@flatbox:/tmp$ sudo apt-get remove -y mergerfs >/dev/null
doak@flatbox:/tmp$ ~/daten/software/linux/mergerfs/build/mergerfs branch merged
warning: mergerfs is not running as root and may not work correctly
doak@flatbox:/tmp$ mount | grep merged
/tmp/branch on /tmp/merged type fuse.mergerfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,default_permissions)

46212 02:07:12.744338 mount("/tmp/mergerfs/branch", ".", "fuse", MS_NOSUID|MS_NODEV, "default_permissions,fd=3,rootmode=40000,user_id=1000,group_id=1000") = -1 EPERM (Operation not permitted) <0.000009>

strace revleals that the mount() system call even fails in case it succeeds (on my machine™). It also differs in the third argument:

doak@flatbox:/tmp$ mount | grep merged
doak@flatbox:/tmp$ strace ~/daten/software/linux/mergerfs/build/mergerfs branch merged 2>&1 | grep mount
mount("/tmp/branch", "/tmp/merged", "fuse.mergerfs", MS_NOSUID|MS_NODEV, "default_permissions,fd=5,rootmod"...) = -1 EPERM (Operation not permitted)
doak@flatbox:/tmp$ mount | grep merged
/tmp/branch on /tmp/merged type fuse.mergerfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,default_permissions)

Honestly, I don't understand why it succeeds eventually. Here is the full strace.log for the successful mount (i.e. mount shows the merged directory):

doak@flatbox:/tmp$ mount | grep merged
doak@flatbox:/tmp$ strace ~/daten/software/linux/mergerfs/build/mergerfs branch merged &>strace.log
doak@flatbox:/tmp$ mount | grep merged
/tmp/branch on /tmp/merged type fuse.mergerfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,default_permissions)

[trapexit]

Without digging in yet...

FUSE filesystems need auxiliary tooling to be mounted as non-root. If you were mixing installed versions then there could be some incompatibility or not setup correctly.

[doak]

Just had a look into mergerfs' readme:

If mergerfs doesn't work as a type it could be due to how the mount.mergerfs tool was installed. Must be in /sbin/ with proper permissions.

Although it makes no sense, since there is no difference regarding the setuid bit in case merferfs have or have not been installed by the distribution, I just want to mention it: the setuid bit is not set for these two binaries:

doak@flatbox:~$ ls -l /sbin/mount.{fuse3,mergerfs}
-rwxr-xr-x 1 root root 18664 Jan 12 16:46 /sbin/mount.fuse3
-rwxr-xr-x 1 root root 14568 Sep  2  2022 /sbin/mount.mergerfs