Android smb clients issue on remove files.

[serj-47]

My mergerfs pool consists of two disks and is used to share both disks over the network using the samba protocol, the pool is mounted as follows:

/srv/ST32000641AS:/srv/WD20SPZX /srv/pool mergerfs cache.files=off,dropcacheonclose=true,category.create=epff,fsname=mergerfs,ignorepponrename=true,minfreespace=10G 0 0

smb.conf:

...
[Storage]
path=/srv/pool/Storage
browseable=yes
writeable=yes
only guest=yes
create mask=0777
directory mask=0777
public=yes

So, samba clients with Windows OS or Linux OS can easily access the pool using the samba protocol, and can create, move and delete files and there are no problems.
But there are some problems with smb clients on Android. Here is a list of file managers for android that I have tried:
-Total Commander, (you need to install a separate plugin to work with smb).
-Material Files
In both file managers can create and move files on the samba server, but when trying to delete an error occurs about the inability to delete a directory or file.
*There is no such problem when sharing a shared directory on a disk without mergerfs.

[trapexit]

Please provide the requested info as defined in the ticket you filled out and in the support section of the page.

[joaopedrolages]

i Have the same issue using mergerfs as OMV 7 plugin
When i delete a file on android (total comander or File manager) using SMB share i get the error :
The Samba 'panic action' script, /usr/share/samba/panic-action,
was called for PID 6018 ().

This means there was a problem with the program, such as a segfault.
However, the executable could not be found for process 6018.
It may have died unexpectedly, or you may not have permission to debug
the process.

First i had the error with NTFS on 2 disks but i deleted them and reformated to BTRFS and i get the same error.

No problem with windows

Once i saw user ADB in SMB diagnotics report.

[trapexit]

As I said above I need details as described in the ticket creation page and the support section of the docs. If the app is dying that's not mergerfs' fault. Nothing mergerfs does should cause an app to die. That would be a bug on their side.

[serj-47]

As described in the mergerfs wiki, I see that there may be problems with deleting or moving files in some file managers that do not follow the rfc when working with smb, or something like that. Therefore, I tried several file managers with a built-in smb client available in android, as a result, the 'amaze file manager' perfectly deletes files from mergerfs share and there are no errors.

[schrock]

I am experiencing the same issue as described in this bug report, so here are the details:

Describe the bug

When attempting to delete files on a Samba share hosting files in a mergerfs volume via the Android Solid Explorer client app, the Samba service crashes, and the client app complains about an unknown error. Despite the crash, the file is deleted from the mergerfs volume. The crash and associated error does not occur when deleting files on a Samba share hosting files in a regular volume.

As reported by @serj-47, the Amaze File Manager app does not cause Samba to crash.

Please be sure to use latest release of mergerfs to ensure the issue still exists. Not your distro's latest but the latest official release.

I am using mergerfs v2.40.2.

To Reproduce

Steps to reproduce the behavior. List all steps to reproduce. All settings.

Please simplify the reproduction as much as possible.

• Unless it is dependenat on multiple branches, use a single branch
• Reproduce with standard tooling if possible (touch,truncate,rm,rmdir,ln,etc.) Having to install 3rd party software will make debugging more difficult.

Relevant portion of /etc/fstab:

/dev/sdb2 /mnt/sdb btrfs defaults 0 0
/dev/sdc2 /mnt/sdc btrfs defaults 0 0
/dev/sdd2 /mnt/sdd btrfs defaults 0 0
/dev/sde2 /mnt/sde btrfs defaults 0 0
/mnt/sd* /mnt/media mergerfs category.create=eppfrd,minfreespace=20G,cache.files=off,dropcacheonclose=true 0 0

Relevant portion of /etc/samba/smb.conf:

[media]
  path = /mnt/media
  browseable = yes
  writable = yes
  read only = no
  valid users = schrock
  guest ok = no

[sdb]
  path = /mnt/sdb
  browseable = yes
  writable = yes
  read only = no
  valid users = schrock
  guest ok = no

Steps to reproduce:

1. On the server, create a test file in the root of the mergerfs volume like so: touch /mnt/media/file0
2. In the Android Solid Explorer app, connect to the Samba share media and delete file0.

Expected behavior

The file should be deleted cleanly without Samba crashing on the server or producing any errors on the client.

System information:

• OS, kernel version: uname -a
  • Linux ruby 6.8.0-40-generic #40-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 5 10:34:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
• Samba version: samaba --version
  • Version 4.19.5-Ubuntu
• mergerfs version: mergerfs -V
  • v2.40.2
• mergerfs settings
  • category.create=eppfrd,minfreespace=20G,cache.files=off,dropcacheonclose=true
• List of drives, filesystems, & sizes:
  • df -h

Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.4G  4.8M  1.4G   1% /run
efivarfs        128K   52K   72K  42% /sys/firmware/efi/efivars
/dev/sda2       232G   17G  215G   8% /
tmpfs           6.8G  4.0K  6.8G   1% /dev/shm
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
b:c:d:e         6.4T  3.4T  3.0T  54% /mnt/media
/dev/sdd2       932G  426G  506G  46% /mnt/sdd
/dev/sdc2       932G  507G  425G  55% /mnt/sdc
/dev/sdb2       932G  467G  465G  51% /mnt/sdb
/dev/sde2       3.7T  2.1T  1.7T  56% /mnt/sde
/dev/sda1       1.1G  6.2M  1.1G   1% /boot/efi
tmpfs           1.4G  116K  1.4G   1% /run/user/114
tmpfs           1.4G  112K  1.4G   1% /run/user/1000

• lsblk -f

NAME   FSTYPE FSVER LABEL                UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0                                                                               0   100% /snap/bare/5
loop1                                                                               0   100% /snap/core22/1380
loop2                                                                               0   100% /snap/core22/1439
loop3                                                                               0   100% /snap/firefox/4650
loop4                                                                               0   100% /snap/firefox/4698
loop5                                                                               0   100% /snap/gnome-42-2204/176
loop6                                                                               0   100% /snap/gtk-common-themes/1535
loop7                                                                               0   100% /snap/snapd/21759
sda
├─sda1 vfat   FAT32                      3E98-649D                                 1G     1% /boot/efi
└─sda2 btrfs                             cfd7182f-e551-4d10-bf97-e61d467d45b6  214.2G     7% /var/snap/firefox/common/host-hunspell
                                                                                             /
sdb
├─sdb1
└─sdb2 btrfs        Samsung 850 EVO 1    6029e008-55ce-a7cc-7b76-5e36e1abb80d  464.8G    50% /mnt/sdb
sdc
├─sdc1
└─sdc2 btrfs        Samsung 850 EVO 2    c9dd44b4-f81b-7d7e-4e16-e7ee75dce608  424.2G    54% /mnt/sdc
sdd
├─sdd1
└─sdd2 btrfs        Samsung 850 EVO 3圃ʥ d30929dc-bc1e-e033-5898-c4a5c8d4a129  505.3G    46% /mnt/sdd
sde
├─sde1
└─sde2 btrfs        Samsung 870 QVO 1    d669a52c-bcbf-4c71-d998-79223e280f61    1.6T    55% /mnt/sde

• A strace of the application having a problem:
  • strace -fvTtt -s 256 -o /tmp/strace.smbd.txt -p <appPID>
    • strace.mergerfs.txt
• strace of mergerfs while app tried to do it's thing:
  • strace -fvTtt -s 256 -o /tmp/strace.mergerfs.txt -p <mergerfsPID>
    • strace.smbd.txt
• Samba log:

[2024/08/13 14:08:22.224846,  0] source3/smbd/fd_handle.c:39(fd_handle_destructor)
  PANIC: assert failed at source3/smbd/fd_handle.c(39): (fh->fd == -1) || (fh->fd == AT_FDCWD)
[2024/08/13 14:08:22.224918,  0] lib/util/fault.c:178(smb_panic_log)
  ===============================================================
[2024/08/13 14:08:22.224937,  0] lib/util/fault.c:179(smb_panic_log)
  INTERNAL ERROR: assert failed: (fh->fd == -1) || (fh->fd == AT_FDCWD) in smbd (smbd[10.6.0.2]) (client [10.6.0.2]) pid 5801 (4.19.5-Ubuntu)
[2024/08/13 14:08:22.224951,  0] lib/util/fault.c:186(smb_panic_log)
  If you are running a recent Samba version, and if you think this problem is not yet fixed in the latest versions, please consider reporting this bug, see https://wiki.samba.org/index.php/Bug_Reporting
[2024/08/13 14:08:22.224963,  0] lib/util/fault.c:191(smb_panic_log)
  ===============================================================
[2024/08/13 14:08:22.224973,  0] lib/util/fault.c:192(smb_panic_log)
  PANIC (pid 5801): assert failed: (fh->fd == -1) || (fh->fd == AT_FDCWD) in 4.19.5-Ubuntu
[2024/08/13 14:08:22.225185,  0] lib/util/fault.c:303(log_stack_trace)
  BACKTRACE: 27 stack frames:
   #0 /usr/lib/x86_64-linux-gnu/samba/libgenrand-samba4.so.0(log_stack_trace+0x37) [0x7773fc0bb517]
   #1 /usr/lib/x86_64-linux-gnu/samba/libgenrand-samba4.so.0(smb_panic+0x15) [0x7773fc0bbd25]
   #2 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(+0x3d083) [0x7773fc479083]
   #3 /lib/x86_64-linux-gnu/libtalloc.so.2(+0x44a0) [0x7773fc05e4a0]
   #4 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(file_free+0xe1) [0x7773fc48eb81]
   #5 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(+0xb5e56) [0x7773fc4f1e56]
   #6 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(smbd_smb2_request_process_close+0x259) [0x7773fc4f2179]
   #7 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(smbd_smb2_request_dispatch+0x1b7f) [0x7773fc4e87bf]
   #8 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(+0xae069) [0x7773fc4ea069]
   #9 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_invoke_fd_handler+0x98) [0x7773fc04de48]
   #10 /lib/x86_64-linux-gnu/libtevent.so.0(+0xefda) [0x7773fc051fda]
   #11 /lib/x86_64-linux-gnu/libtevent.so.0(+0x6004) [0x7773fc049004]
   #12 /lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x9b) [0x7773fc04abab]
   #13 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x2b) [0x7773fc04acfb]
   #14 /lib/x86_64-linux-gnu/libtevent.so.0(+0x6084) [0x7773fc049084]
   #15 /usr/lib/x86_64-linux-gnu/samba/libsmbd-base-samba4.so.0(smbd_process+0x7eb) [0x7773fc4d534b]
   #16 smbd: client [10.6.0.2](+0xa5d6) [0x5dbb808d45d6]
   #17 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_invoke_fd_handler+0x98) [0x7773fc04de48]
   #18 /lib/x86_64-linux-gnu/libtevent.so.0(+0xefda) [0x7773fc051fda]
   #19 /lib/x86_64-linux-gnu/libtevent.so.0(+0x6004) [0x7773fc049004]
   #20 /lib/x86_64-linux-gnu/libtevent.so.0(_tevent_loop_once+0x9b) [0x7773fc04abab]
   #21 /lib/x86_64-linux-gnu/libtevent.so.0(tevent_common_loop_wait+0x2b) [0x7773fc04acfb]
   #22 /lib/x86_64-linux-gnu/libtevent.so.0(+0x6084) [0x7773fc049084]
   #23 smbd: client [10.6.0.2](main+0x1432) [0x5dbb808d2032]
   #24 /lib/x86_64-linux-gnu/libc.so.6(+0x2a1ca) [0x7773fbe2a1ca]
   #25 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x8b) [0x7773fbe2a28b]
   #26 smbd: client [10.6.0.2](_start+0x25) [0x5dbb808d2a95]
[2024/08/13 14:08:22.225325,  0] source3/lib/util.c:691(smb_panic_s3)
  smb_panic(): calling panic action [/usr/share/samba/panic-action 5801]
[2024/08/13 14:08:22.226736,  0] source3/lib/util.c:698(smb_panic_s3)
  smb_panic(): action returned status 0
[2024/08/13 14:08:22.226765,  0] source3/lib/dumpcore.c:317(dump_core)
  coredump is handled by helper binary specified at /proc/sys/kernel/core_pattern

[trapexit]

Thank you for the detailed report but samba is crashing itself. This really should be a report to them. Even if mergerfs was returning something odd it shouldn't panic. And looking at the strace shows smbd unlinking "file0" and then closing the fd. Both succeed and look fine. I just tried using Solid Explorer to create and delete a file using samba 4.19.6. No crash.

BTW, your traces are from different times. Your mergerfs trace is minutes after the smbd one.

[schrock]

Thank you for looking at the log files. The thing I find interesting is that I have two Samba shares, one for a regular volume and another for the mergerfs volume. The Samba crash only occurs when deleting files from the mergerfs volume, not the regular volume, which makes me think mergerfs is doing something differently, triggering a different code path in Samba and causing it to crash. I will do some more debugging on my side and see if I can get Samba to log more information.

[trapexit]

At the end of the day a filesystem function is pretty straight forward in that it returns a standard Unix "return int or -1 with errno" and does the thing mentioned. And all of this goes through the kernel which has lots of validation checks. The only "different" thing mergerfs can be doing really is the filesystem behavior itself which of course could cause issues but those are Samba issues in that it is expecting something that may or many not be valid. It certainly shouldn't be issuing a panic because it got confused. Lots of filesystems are not fully POSIX compliant and should at most error gracefully.

I'm not saying this isn't something that could be addressed by mergerfs. What I'm saying is that step one is to understand why Samba panics and kills itself for something that simply can not worth doing that over. That trace from the log isn't detailed enough for me to guess why that is. You have to remember that Samba is complex and it isn't as simple as "you say delete files, samba issues delete, mergerfs delete." There could be hundreds of other syscalls and things going on otherwise.

[Gloryandel]

Although the results returned by different samba clients may fail when deleting files, in reality, it was successful and the files were deleted. Everything is normal on Windows. Samba did not report any errors.
ES indicates failure, Mix indicates success.
samba: ArchLinux samba 2:4.21.1-1

[trapexit]

Again... really need to know what the specifics are of the error from the app's perspective or why samba panics.