You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sometimes "podman image load --input file.tar" hangs indefinitely. It happens inside a privileged podman container on s390x rarely.
Steps to reproduce the issue
Steps to reproduce the issue
podman image load --input file.tar
Describe the results you received
Sometimes command never starts loading image and never finishes.
Describe the results you expected
Podman should successfully load image in a reasonable amount of time.
podman info output
$ podman version Client: Podman Engine Version: 4.9.3 API Version: 4.9.3 Go Version: go1.22.2 Built: Thu Jan 1 00:00:00 1970 OS/Arch: linux/s390x $ podman info host:
arch: s390x buildahVersion: 1.33.7 cgroupControllers:
- cpuset
- cpu
- io
- memory
- hugetlb
- pids cgroupManager: cgroupfs cgroupVersion: v2conmon:
package: conmon_2.1.10+ds1-1build2_s390x path: /usr/bin/conmon version: 'conmon version 2.1.10, commit: unknown'cpuUtilization:
idlePercent: 93.88systemPercent: 0.53userPercent: 5.59cpus: 6databaseBackend: sqlite distribution:
codename: noble distribution: ubuntu version: "24.04"eventLogger: file freeLocks: 2048hostname: f69a1e05b4cd idMappings:
gidmap:
- container_id: 0host_id: 1001size: 1
- container_id: 1host_id: 165536size: 65536uidmap:
- container_id: 0host_id: 1001size: 1
- container_id: 1host_id: 165536size: 65536kernel: 5.14.0-362.8.1.el9_3.s390xlinkmode: dynamiclogDriver: k8s-filememFree: 7710609408memTotal: 25023684608networkBackend: cninetworkBackendInfo:
backend: cnidns: {}package: containernetworking-plugins_1.1.1+ds1-3ubuntu0.24.04.2_s390xpath: /usr/lib/cniociRuntime:
name: crunpackage: crun_1.14.1-1_s390xpath: /usr/bin/crunversion: |- crun version 1.14.1 commit: de537a7965bfbe9992e2cfae0baeb56a08128171 rundir: /tmp/podman-run-1001/crun spec: 1.0.0 +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJLos: linuxpasta:
executable: /usr/bin/pasta package: passt_0.0~git20240220.1e6f92b-1_s390x version: | pasta unknown version Copyright Red Hat GNU General Public License, version 2 or later <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. remoteSocket:
exists: falsepath: /tmp/podman-run-1001/podman/podman.socksecurity:
apparmorEnabled: falsecapabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOTrootless: trueseccompEnabled: trueseccompProfilePath: /usr/share/containers/seccomp.jsonselinuxEnabled: falseserviceIsRemote: falseslirp4netns:
executable: /usr/bin/slirp4netnspackage: slirp4netns_1.2.1-1build2_s390xversion: |- slirp4netns version 1.2.1 commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194 libslirp: 4.7.0 SLIRP_CONFIG_VERSION_MAX: 4 libseccomp: 2.5.5swapFree: 0swapTotal: 0uptime: 697h 30m 20.00s (Approximately 29.04 days)variant: ""plugins:
authorization: nulllog:
- k8s-file
- none
- passthrough
- journaldnetwork:
- bridge
- macvlan
- ipvlanvolume:
- localregistries:
search:
- docker.iostore:
configFile: /home/actions-runner/.config/containers/storage.confcontainerStore:
number: 0paused: 0running: 0stopped: 0graphDriverName: overlaygraphOptions: {}graphRoot: /home/actions-runner/.local/share/containers/storagegraphRootAllocated: 214212521984graphRootUsed: 69358649344graphStatus:
Backing Filesystem: overlayfsNative Overlay Diff: "false"Supports d_type: "true"Supports shifting: "true"Supports volatile: "true"Using metacopy: "false"imageCopyTmpDir: /var/tmpimageStore:
number: 0runRoot: /tmp/containers-user-1001/containerstransientStore: falsevolumePath: /home/actions-runner/.local/share/containers/storage/volumesversion:
APIVersion: 4.9.3Built: 0BuiltTime: Thu Jan 1 00:00:00 1970GitCommit: ""GoVersion: go1.22.2Os: linuxOsArch: linux/s390xVersion: 4.9.3$ dpkg-query -l podmanDesired=Unknown/Install/Remove/Purge/Hold| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)||/ Name Version Architecture Description+++-==============-====================-============-==================================ii podman 4.9.3+ds1-1ubuntu0.2 s390x tool to manage containers and pods
Podman in a container
Yes
Privileged Or Rootless
Privileged
Upstream Latest Release
No
Additional environment details
Host is RHEL 9.5 s390x in a cloud, inside KVM. It runs podman as root, guest is privileged podman container with Ubuntu 24.04. Issue reproduces in this Ubuntu system.
Additional information
Backtrace of child process hanging:
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x3ff4e51a8c0 (LWP 18) "podman" futex_wait (private=0, expected=2, futex_word=0x3ff48000030) at ../sysdeps/nptl/futex-internal.h:146
(gdb) bt
#0 futex_wait (private=0, expected=2, futex_word=0x3ff48000030) at ../sysdeps/nptl/futex-internal.h:146
#1 __GI___lll_lock_wait_private (futex=futex@entry=0x3ff48000030) at lowlevellock.c:34
#2 0x000003ff97eb0b40 in __malloc_fork_lock_parent () at arena.c:180
#3 0x000003ff97eec170 in __libc_fork () at fork.c:71
#4 0x00000000023f93e8 in create_pause_process (pause_pid_file_path=pause_pid_file_path@entry=0x3ff3002f9f0 "/tmp/podman-run-1001/libpod/tmp/pause.pid", argv=0x3ff30029340,
argv@entry=<error reading variable: value has been optimized out>) at rootless_linux.c:661
#5 0x00000000023fa322 in reexec_in_user_namespace (ready=7, pause_pid_file_path=0x3ff3002f9f0 "/tmp/podman-run-1001/libpod/tmp/pause.pid", file_to_read=<optimized out>, outputfd=0)
at rootless_linux.c:1120
#6 0x00000000023f8ae2 in _cgo_ed50cdecc5c8_Cfunc_reexec_in_user_namespace (v=0xc0004748e0) at cgo-gcc-prolog:92
#7 0x000000000109fdf8 in runtime.asmcgocall () at runtime/asm_s390x.s:549
#8 0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
It looks like podman first does syscall(clone) [1] and then does fork glibc function call [2].
According to what @iii-i told me, calling fork glibc function should be avoided after doing syscall(clone(~CLONE_VM)) or similar actions, because glibc internal state may be inconsistent, and that can lead to fork function hanging.
Either fork calls should be replaced by syscall(clone) reimplementations to avoid going inside glibc fork if they may be called after doing syscall(clone), or syscall(clone) could be reimplemented as compination of fork+unshare, or even just unshare if fork is not needed.
Issue Description
Sometimes "podman image load --input file.tar" hangs indefinitely. It happens inside a privileged podman container on s390x rarely.
Steps to reproduce the issue
Steps to reproduce the issue
Describe the results you received
Sometimes command never starts loading image and never finishes.
Describe the results you expected
Podman should successfully load image in a reasonable amount of time.
podman info output
Podman in a container
Yes
Privileged Or Rootless
Privileged
Upstream Latest Release
No
Additional environment details
Host is RHEL 9.5 s390x in a cloud, inside KVM. It runs podman as root, guest is privileged podman container with Ubuntu 24.04. Issue reproduces in this Ubuntu system.
Additional information
Backtrace of child process hanging:
It looks like podman first does
syscall(clone)
[1] and then doesfork
glibc function call [2].According to what @iii-i told me, calling
fork
glibc function should be avoided after doingsyscall(clone(~CLONE_VM))
or similar actions, because glibc internal state may be inconsistent, and that can lead tofork
function hanging.Either
fork
calls should be replaced bysyscall(clone)
reimplementations to avoid going inside glibcfork
if they may be called after doingsyscall(clone)
, orsyscall(clone)
could be reimplemented as compination offork
+unshare
, or even justunshare
iffork
is not needed.[1]
podman/pkg/rootless/rootless_linux.c
Line 973 in 48f8742
[2]
podman/pkg/rootless/rootless_linux.c
Line 661 in 48f8742
The text was updated successfully, but these errors were encountered: