cannot use nix --store runtime which is selected by default

[orzFly]

Using nix-portable v011

$ nix-portable nix run nixpkgs#htop
error: setting up a private mount namespace: Operation not permitted

By the way, bwrap runs fine.

$ NP_RUNTIME=bwrap nix-portable nix run nixpkgs#htop

Diagnostics

$ zgrep CONFIG_USER_NS /proc/config.gz
CONFIG_USER_NS=y
CONFIG_USER_NS_UNPRIVILEGED=y

$ cat /proc/sys/kernel/unprivileged_userns_clone
1

$ unshare -r -n echo YES
YES

$ uname -a
Linux mipha 6.7.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 00:30:36 +0000 x86_64 GNU/Linux

$ NP_DEBUG=2 nix-portable nix run nixpkgs#htop 2>&1 | tee np_debug.log

np_debug.log

$ NP_DEBUG=2 strace nix-portable nix run nixpkgs#htop 2>&1 | tee strace.log

strace.log

$ grep unshare strace.log
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)

$ strace unshare -r -m echo YES 2>&1 | egrep 'unshare|YES'
execve("/usr/bin/unshare", ["unshare", "-r", "-m", "echo", "YES"], 0x7ffc6e302bd0 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
execve("/home/user/perl5/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/opt/asdf-vm/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/home/user/sbin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/home/user/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/local/sbin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/local/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = -1 ENOENT (No such file or directory)
execve("/usr/bin/echo", ["echo", "YES"], 0x7ffdf5a71d28 /* 79 vars */) = 0
write(1, "YES\n", 4YES

[DavHau]

Thanks for the detailed report. I'm not sure at the moment why the nix local store doesn't work but bubblewrap does.

Anyways, v012 should fix the issue with nix-portable not falling back to bwrap automatically. Can you confirm that it works now?

[orzFly]

Anyways, v012 should fix the issue with nix-portable not falling back to bwrap automatically. Can you confirm that it works now?

Still no luck 😿 same error

I updated nix-portable, nuked ~/.nix-portable and tried:

$ nix-portable nix run nixpkgs#htop
error: setting up a private mount namespace: Operation not permitted

Removing last_auto_runtime for a completer log

$ rm /home/user/.nix-portable/conf/last_auto_runtime
$ NP_DEBUG=2 nix-portable nix run nixpkgs#htop >np_debug_v012.log 2>&1

np_debug_v012.log

[orzFly]

Worth noting in np_debug_v012.log:

+ /home/user/.nix-portable/bin/nix --store /home/user/.nix-portable/tmp/__store shell -f /home/user/.nix-portable/mini-drv.nix -c /home/user/.nix-portable/bin/nix store add-file --store /home/user/.nix-portable/tmp/__store /home/user/.nix-portable/tmp/testfile
this derivation will be built:
  /nix/store/mbwnbda8s0f5rlpprm2km8m69gkq94nm-foo.drv
building '/nix/store/mbwnbda8s0f5rlpprm2km8m69gkq94nm-foo.drv'...
+ chmod -R +w /home/user/.nix-portable/tmp/__store
+ rm -r /home/user/.nix-portable/tmp/__store
+ debug 'nix --store works on this system -> will use nix as runtime'
+ echo nix --store works on this system '->' will use nix as runtime
nix --store works on this system -> will use nix as runtime

I am not sure why the second call to unshare is failed. I am not familiar with unshare-y things.

$ strace nix-portable nix run nixpkgs#htop 2>&1 | grep unshare
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)

This command can success. I am not sure if this helps.

$ strace /usr/bin/unshare -r -m /usr/bin/unshare -r -m echo YES 2>&1 | grep unshare
execve("/usr/bin/unshare", ["/usr/bin/unshare", "-r", "-m", "/usr/bin/unshare", "-r", "-m", "echo", "YES"], 0x7ffcfe206b78 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
execve("/usr/bin/unshare", ["/usr/bin/unshare", "-r", "-m", "echo", "YES"], 0x7ffce18616e0 /* 79 vars */) = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0

[DavHau]

Which version of which distro are you running? Maybe I can add a test for it to the CI pipeline.

[orzFly]

Arch Linux. Up-to-dated (with an old kernel for ZFS).

---

However, this is a little embarrassing to say... Actually, I am new to nix. Due to my oversatisfaction to bundling applications with nix-portable as single executable, I just installed Nix (multiple users install)! With the real /nix directory, things has changed.

$ uname -a
Linux mipha 6.7.8-arch1-1 #1 SMP PREEMPT_DYNAMIC Sun, 03 Mar 2024 00:30:36 +0000 x86_64 GNU/Linux

$ sudo zfs unmount rpool/nix
$ mount | grep /nix
$ ls -alh /nix
total 10K
drwxr-xr-x  2 root root  2 Apr 16 02:10 .
drwxr-xr-x 18 root root 24 Apr 16 02:10 ..
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs#hello 2>&1 | egrep unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)
write(2, "error: setting up a private moun"..., 69error: setting up a private mount namespace: Operation not permitted

$ sudo zfs mount rpool/nix
$ mount | grep /nix
rpool/nix on /nix type zfs (rw,relatime,xattr,posixacl,casesensitive)
$ ls -alh /nix
total 1.1M
drwxr-xr-x   4 root root      4 Apr 16 02:11 .
drwxr-xr-x  18 root root     24 Apr 16 02:10 ..
drwxrwxr-t 461 root nixbld 4.2K Apr 16 02:27 store
drwxr-xr-x   4 root root      4 Apr 16 02:14 var
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs#hello 2>&1 | egrep unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
write(1, "Hello, world!\n", 14Hello, world!

[orzFly]

Reproducible with ArchLinux Netboot Release 2024.04.01. Here's the rough session transcript:

# uname -a
Linux archiso 6.8.2-arch2-1 #1 SMP PREEMPT_DYNAMIC Thu, 28 Mar 2024 17:06:35 +0000 x86_64 GNU/Linux
# mount -o remount,size=50% /run/archiso/cowspace
# pacman -Sy wget strace
# useradd user
# mkdir /home/user
# chown -R user:user /home/user
# sudo -uuser -Hs
$ wget https://github.com/DavHau/nix-portable/releases/download/v012/nix-portable-x86_64
$ chmod +x nix-portable-x86_64
$ ./nix-portable-x86_64 nix run nixpkgs\#hello                          # let it download and fail
$ NP_RUNTIME=nix NP_DEBUG=2 strace ./nix-portable-x86_64 nix run nixpkgs\#hello 2>&1 | grep -E unshare\|error:\|Hello
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = 0
unshare(CLONE_NEWNS|CLONE_NEWUSER)      = -1 EPERM (Operation not permitted)
unshare(CLONE_NEWNS)                    = -1 EPERM (Operation not permitted)
write(2, "error: setting up a private moun"..., 69error: setting up a private mount namespace: Operation not permitted
$ NP_RUNTIME=bwrap ./nix-portable-x86_64 nix run nixpkgs\#hello
Hello, world!

[orzFly]

Maybe I can add a test for it to the CI pipeline.

Maybe you should try adding test for nix run nixpkgs#hello to the CI pipeline...

Producible on Debian GNU/Linux 11 (bullseye)

orzfly@media-sorter:~$ NP_RUNTIME=bwrap ./nix-portable nix-shell -p hello --run hello
Hello, world!

orzfly@media-sorter:~$ NP_RUNTIME=bwrap ./nix-portable nix run nixpkgs#hello
Hello, world!

orzfly@media-sorter:~$ NP_RUNTIME=nix ./nix-portable nix-shell -p hello --run hello
Hello, world!

orzfly@media-sorter:~$ NP_RUNTIME=nix ./nix-portable nix run nixpkgs#hello
error: setting up a private mount namespace: Operation not permitted

orzfly@media-sorter:~$ uname -a
Linux media-sorter 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.37-1 (2023-07-03) x86_64 GNU/Linux

orzfly@media-sorter:~$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

[ccornix]

Reproducible on Debian GNU/Linux 12 (bookworm) as well, using v012.

ccornix@debian:~$ NP_RUNTIME=bwrap ./nix-portable nix run nixpkgs#hello
Hello, world!

ccornix@debian:~$ NP_RUNTIME=nix ./nix-portable nix run nixpkgs#hello
error: setting up a private mount namespace: Operation not permitted

ccornix@debian:~$ uname -a
Linux debian 6.1.0-20-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.85-1 (2024-04-11) x86_64 GNU/Linux

ccornix@debian:~$ lsb_release -a
No LSB modules are available.
Distributor ID:	Debian
Description:	Debian GNU/Linux 12 (bookworm)
Release:	12
Codename:	bookworm

[chaserhkj]

Considering NixOS/nix#6853 is present and probably will be around for some time as well. Maybe we should just switch to bwrap runtime as default in nix-portable ?