Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

podman image load --input file.tar sometimes hangs infinitely #25184

Open
AlekseiNikiforovIBM opened this issue Jan 31, 2025 · 2 comments
Open
Labels
kind/bug Categorizes issue or PR as related to a bug.

Comments

@AlekseiNikiforovIBM
Copy link

AlekseiNikiforovIBM commented Jan 31, 2025

Issue Description

Sometimes "podman image load --input file.tar" hangs indefinitely. It happens inside a privileged podman container on s390x rarely.

Steps to reproduce the issue

Steps to reproduce the issue

  1. podman image load --input file.tar

Describe the results you received

Sometimes command never starts loading image and never finishes.

Describe the results you expected

Podman should successfully load image in a reasonable amount of time.

podman info output

$ podman version                                                                                                                                                                
Client:       Podman Engine                                                                                                                                                                                  
Version:      4.9.3                                                                                                                                                                                          
API Version:  4.9.3                                                                                                                                                                                          
Go Version:   go1.22.2                                                                                                                                                                                       
Built:        Thu Jan  1 00:00:00 1970                                                                                                                                                                       
OS/Arch:      linux/s390x   
$ podman info                                                                                                                                                                   
host:                                                                                                                                                                                                        
  arch: s390x                                                                                                                                                                                                
  buildahVersion: 1.33.7                                                                                                                                                                                     
  cgroupControllers:                                                                                                                                                                                         
  - cpuset                                                                                                                                                                                                   
  - cpu                                                                                                                                                                                                      
  - io                                                                                                                                                                                                       
  - memory                                                                                                                                                                                                   
  - hugetlb                                                                                                                                                                                                  
  - pids                                                                                                                                                                                                     
  cgroupManager: cgroupfs                                                                                                                                                                                    
  cgroupVersion: v2                                                                                                                                                                                          
  conmon:                                                                                                                                                                                                    
    package: conmon_2.1.10+ds1-1build2_s390x                                                                                                                                                                 
    path: /usr/bin/conmon                                                                                                                                                                                    
    version: 'conmon version 2.1.10, commit: unknown'                                                                                                                                                        
  cpuUtilization:                                                                                                                                                                                            
    idlePercent: 93.88                                                                                                                                                                                       
    systemPercent: 0.53                                                                                                                                                                                      
    userPercent: 5.59                                                                                                                                                                                        
  cpus: 6                                                                                                                                                                                                    
  databaseBackend: sqlite                                                                                                                                                                                    
  distribution:                                                                                                                                                                                              
    codename: noble                                                                                                                                                                                          
    distribution: ubuntu                                                                                                                                                                                     
    version: "24.04"                                                                                                                                                                                         
  eventLogger: file                                                                                                                                                                                          
  freeLocks: 2048                                                                                                                                                                                            
  hostname: f69a1e05b4cd                                                                                                                                                                                     
  idMappings:                   
    gidmap:                                                                                                                                                                                                  
    - container_id: 0                                                                                                                                                                                        
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.14.0-362.8.1.el9_3.s390x
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 7710609408
  memTotal: 25023684608
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns: {}
    package: containernetworking-plugins_1.1.1+ds1-3ubuntu0.24.04.2_s390x
    path: /usr/lib/cni
  ociRuntime:
    name: crun
    package: crun_1.14.1-1_s390x
    path: /usr/bin/crun
    version: |-
      crun version 1.14.1
      commit: de537a7965bfbe9992e2cfae0baeb56a08128171
      rundir: /tmp/podman-run-1001/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  pasta:                                                                                                                                                                                                     
    executable: /usr/bin/pasta                                                                                                                                                                               
    package: passt_0.0~git20240220.1e6f92b-1_s390x                                                                                                                                                           
    version: |                                                                                                                                                                                               
      pasta unknown version                                                                                                                                                                                  
      Copyright Red Hat                                                                                                                                                                                      
      GNU General Public License, version 2 or later                                                                                                                                                         
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>                                                                                                                                             
      This is free software: you are free to change and redistribute it.                                                                                                                                     
      There is NO WARRANTY, to the extent permitted by law.                                                                                                                                                  
  remoteSocket:                                                                                                                                                                                              
    exists: false
    path: /tmp/podman-run-1001/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns_1.2.1-1build2_s390x
    version: |-
      slirp4netns version 1.2.1
      commit: 09e31e92fa3d2a1d3ca261adaeb012c8d75a8194
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 697h 30m 20.00s (Approximately 29.04 days)
  variant: ""
plugins:                                                                                                                                                                                                     
  authorization: null                                                                                                                                                                                        
  log:                                                                                                                                                                                                       
  - k8s-file                                                                                                                                                                                                 
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /home/actions-runner/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /home/actions-runner/.local/share/containers/storage
  graphRootAllocated: 214212521984
  graphRootUsed: 69358649344
  graphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /tmp/containers-user-1001/containers
  transientStore: false
  volumePath: /home/actions-runner/.local/share/containers/storage/volumes
version:
  APIVersion: 4.9.3
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/s390x
  Version: 4.9.3
$ dpkg-query -l podman
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version              Architecture Description
+++-==============-====================-============-==================================
ii  podman         4.9.3+ds1-1ubuntu0.2 s390x        tool to manage containers and pods

Podman in a container

Yes

Privileged Or Rootless

Privileged

Upstream Latest Release

No

Additional environment details

Host is RHEL 9.5 s390x in a cloud, inside KVM. It runs podman as root, guest is privileged podman container with Ubuntu 24.04. Issue reproduces in this Ubuntu system.

Additional information

Backtrace of child process hanging:

(gdb) info threads
  Id   Target Id                              Frame
* 1    Thread 0x3ff4e51a8c0 (LWP 18) "podman" futex_wait (private=0, expected=2, futex_word=0x3ff48000030) at ../sysdeps/nptl/futex-internal.h:146
(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x3ff48000030) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait_private (futex=futex@entry=0x3ff48000030) at lowlevellock.c:34
#2  0x000003ff97eb0b40 in __malloc_fork_lock_parent () at arena.c:180
#3  0x000003ff97eec170 in __libc_fork () at fork.c:71
#4  0x00000000023f93e8 in create_pause_process (pause_pid_file_path=pause_pid_file_path@entry=0x3ff3002f9f0 "/tmp/podman-run-1001/libpod/tmp/pause.pid", argv=0x3ff30029340,
    argv@entry=<error reading variable: value has been optimized out>) at rootless_linux.c:661
#5  0x00000000023fa322 in reexec_in_user_namespace (ready=7, pause_pid_file_path=0x3ff3002f9f0 "/tmp/podman-run-1001/libpod/tmp/pause.pid", file_to_read=<optimized out>, outputfd=0)
    at rootless_linux.c:1120
#6  0x00000000023f8ae2 in _cgo_ed50cdecc5c8_Cfunc_reexec_in_user_namespace (v=0xc0004748e0) at cgo-gcc-prolog:92
#7  0x000000000109fdf8 in runtime.asmcgocall () at runtime/asm_s390x.s:549
#8  0x0000000000000000 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

It looks like podman first does syscall(clone) [1] and then does fork glibc function call [2].
According to what @iii-i told me, calling fork glibc function should be avoided after doing syscall(clone(~CLONE_VM)) or similar actions, because glibc internal state may be inconsistent, and that can lead to fork function hanging.

Either fork calls should be replaced by syscall(clone) reimplementations to avoid going inside glibc fork if they may be called after doing syscall(clone), or syscall(clone) could be reimplemented as compination of fork+unshare, or even just unshare if fork is not needed.

[1]

pid = syscall_clone (CLONE_NEWUSER|CLONE_NEWNS|SIGCHLD, NULL);

[2]
pid = fork ();

@AlekseiNikiforovIBM AlekseiNikiforovIBM added the kind/bug Categorizes issue or PR as related to a bug. label Jan 31, 2025
@Luap99
Copy link
Member

Luap99 commented Jan 31, 2025

@giuseppe PTAL

@giuseppe
Copy link
Member

giuseppe commented Feb 3, 2025

thanks for the report.

Could we replace the fork instead? Something like:

diff --git a/pkg/rootless/rootless_linux.c b/pkg/rootless/rootless_linux.c
index 4f71d49e5c..3d74af6a6c 100644
--- a/pkg/rootless/rootless_linux.c
+++ b/pkg/rootless/rootless_linux.c
@@ -658,7 +658,7 @@ create_pause_process (const char *pause_pid_file_path, char **argv)
   if (pipe (p) < 0)
     return -1;

-  pid = fork ();
+  pid = syscall_clone (SIGCHLD, NULL);
   if (pid < 0)
     {
       close (p[0]);
@@ -689,7 +689,7 @@ create_pause_process (const char *pause_pid_file_path, char **argv)
       close (p[0]);

       setsid ();
-      pid = fork ();
+      pid = syscall_clone (SIGCHLD, NULL);
       if (pid < 0)
         _exit (EXIT_FAILURE);

I am not able to reproduce locally on x64_64, are you able to test this patch?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug.
Projects
None yet
Development

No branches or pull requests

3 participants