Cleanup after podman build takes forever with btrfs driver

[olhovpav]

Issue Description

I’m building a Buildroot-based OS using podman build with the btrfs storage driver. My Dockerfile has a lot of layers with different invocations of make <package>. During build, a lot of layers produce millions of files in each layer.

After each build, podman prints the message Successfully tagged ... and then hangs. The build process up to the message takes about two hours for full build and seconds when using cache. However, the phase after the success message always takes hours, much longer than the build itself.

In the middle of this hang-up, I started strace -f until the completion of the podman build command. Here are some summary stats:

wc -l strace-output.txt
129135582 /home/user/strace-output.txt

du -hs strace-output.txt
7.8G /home/user/strace-output.txt

cat strace-output.txt | grep newfstatat | wc -l
2328635

cat strace-output.txt | grep openat | wc -l
4808612

cat strace-output.txt | grep getdents64 | wc -l
9787523

cat strace-output.txt | grep nanosleep | wc -l
46762207

cat strace-output.txt | grep unlinkat | wc -l
58091993

About 36% of the syscalls are nanosleep, and 45% are unlinkat at $graphRoot/btrfs/subvolumes/**. I assume it's all related to this cleanup code path.

Steps to reproduce the issue

1. Enable the btrfs storage driver.
2. Build a large image with many layers and millions of files per layer.
3. Wait for "Successfully tagged …".
4. Observe that the process hangs for hours.

Describe the results you received

The cleanup process takes much longer than the build itself, even when there was no real build and all layers were taken from cache.

Describe the results you expected

I expect it not to take so much time to finish the cleanup process, especially when all layers were taken from cache. If using cache, it takes a second to create all the layers, I expect it shouldn't spend hours hours cleaning them up.

podman info output

podman info:
host:
  arch: amd64
  buildahVersion: 1.40.1
  cgroupControllers:
  - cpu
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-1:2.1.13-1
    path: /usr/bin/conmon
    version: 'conmon version 2.1.13, commit: 82de887596ed8ee6d9b2ee85e4f167f307bb569b'
  cpuUtilization:
    idlePercent: 90.71
    systemPercent: 4.32
    userPercent: 4.96
  cpus: 16
  databaseBackend: boltdb
  distribution:
    distribution: manjaro
    version: unknown
  eventLogger: journald
  freeLocks: 2040
  hostname: tuf
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    - container_id: 65537
      host_id: 200000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 6.15.9-2-MANJARO
  linkmode: dynamic
  logDriver: journald
  memFree: 591417344
  memTotal: 32847630336
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.15.0-1
      path: /usr/lib/podman/aardvark-dns
      version: aardvark-dns 1.15.0
    package: netavark-1.15.2-1
    path: /usr/lib/podman/netavark
    version: netavark 1.15.2
  ociRuntime:
    name: crun
    package: crun-1.23.1-1
    path: /usr/bin/crun
    version: |-
      crun version 1.23.1
      commit: d20b23dba05e822b93b82f2f34fd5dada433e0c2
      rundir: /run/user/1000/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-2025_06_11.0293c6f-1
    version: |
      pasta 2025_06_11.0293c6f
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /etc/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.3.3-1
    version: |-
      slirp4netns version 1.3.3
      commit: 944fa94090e1fd1312232cbc0e6b43585553d824
      libslirp: 4.9.1
      SLIRP_CONFIG_VERSION_MAX: 6
      libseccomp: 2.5.6
  swapFree: 26152345600
  swapTotal: 42949668864
  uptime: 245h 33m 25.00s (Approximately 10.21 days)
  variant: ""
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries: {}
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 85
    paused: 0
    running: 5
    stopped: 80
  graphDriverName: btrfs
  graphOptions: {}
  graphRoot: /home/user/data2/podman-storage
  graphRootAllocated: 1706609868800
  graphRootUsed: 1601347534848
  graphStatus:
    Build Version: Btrfs v6.15
    Library Version: "104"
  imageCopyTmpDir: /home/user/data2/podman-tmp
  imageStore:
    number: 1157
  runRoot: /home/user/data2/podman-storage/temp
  transientStore: false
  volumePath: /home/user/data2/podman-storage/volumes
version:
  APIVersion: 5.5.2
  Built: 1751012144
  BuiltTime: Fri Jun 27 11:15:44 2025
  GitCommit: e7d8226745ba07a64b7176a7f128e4ef53225a0e
  GoVersion: go1.24.4
  Os: linux
  OsArch: linux/amd64
  Version: 5.5.2

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

No response

Additional information

No response

[mheon]

The btrfs storage driver is basically best-effort support at this point (overlay on top of a btr filesystem works better in most cases), but I'll ask @flouthoc if he's ever seen anything like this.