Podman imagestore set, but new images still in graphroot

[SoloGao]

Since no response in podman repo, I have to post it again here cuz it's storage-related.

Issue Description

I have an imagestore directory for rootless users to store their images. However, when I use podman pull, only graphroot increases volume size.

Steps to reproduce the issue

Steps to reproduce the issue

1. set imagestore
2. podman system migrate
3. pull a new image
4. watch the increase in size for graphroot and imagestore

Describe the results you received

graphroot increases about the image size
imagestore still less than 1MB

Describe the results you expected

graphroot keeps its size
imagestore increases size

podman info output

host:
  arch: amd64
  buildahVersion: 1.33.5
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.module+el8.10.0+21077+98b84d8a.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: 80c4f656297773fb630a4d966add3242abab39a4'
  cpuUtilization:
    idlePercent: 99.99
    systemPercent: 0.01
    userPercent: 0.01
  cpus: 256
  databaseBackend: sqlite
  distribution:
    distribution: rhel
    version: "8.10"
  eventLogger: journald
  freeLocks: 2048
  hostname: ##HOSTNAME##
  idMappings:
    gidmap:
    - container_id: 0
      host_id: ##USER_GID##
      size: 1
    - container_id: 1
      host_id: 8388608
      size: 65536
    uidmap:
    - container_id: 0
      host_id: ##USER_UID##
      size: 1
    - container_id: 1
      host_id: 8388608
      size: 65536
  kernel: 4.18.0-553.el8_10.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 2156602961920
  memTotal: 2163771584512
  networkBackend: cni
  networkBackendInfo:
    backend: cni
    dns:
      package: podman-plugins-4.9.4-0.1.module+el8.10.0+21350+ea09fba1.x86_64
      path: /usr/libexec/cni/dnsname
      version: |-
        CNI dnsname plugin
        version: 1.4.0-dev
        commit: unknown
        CNI protocol versions supported: 0.1.0, 0.2.0, 0.3.0, 0.3.1, 0.4.0, 1.0.0
    package: containernetworking-plugins-1.4.0-2.module+el8.10.0+21366+f9cb49f8.x86_64
    path: /usr/libexec/cni
  ociRuntime:
    name: crun
    package: crun-1.14.3-2.module+el8.10.0+21340+c6c7475a.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.3
      commit: 1961d211ba98f532ea52d2e80f4c20359f241a98
      rundir: /run/user/##USER_UID##/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux  
  pasta:
    executable: ""
    package: ""
    version: ""
  remoteSocket:
    exists: false
    path: /run/user/##USER_UID##/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.3-1.module+el8.10.0+21306+6be40ce7.x86_64
    version: |-
      slirp4netns version 1.2.3
      commit: c22fde291bb35b354e6ca44d13be181c76a0a432
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 68719472640
  swapTotal: 68719472640
  uptime: 206h 47m 44.00s (Approximately 8.58 days)
  variant: ""plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /##USER_HOME##/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay  
  graphOptions:
    overlay.force_mask: private
    overlay.imagestore: /proj/podman/##USER_NAME##/images
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.13-1.module+el8.10.0+20412+95ee28e2.x86_64
      Version: |-
        fusermount3 version: 3.3.0
        fuse-overlayfs: version 1.13-dev
        FUSE library version 3.3.0
        using FUSE kernel interface version 7.26
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /tmp/podman/##USER_NAME##
  graphRootAllocated: 931028791296
  graphRootUsed: 6790819840
  graphStatus:
    Backing Filesystem: nfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Supports shifting: "true"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 0
  runRoot: /run/user/##USER_UID##/containers
  transientStore: false
  volumePath: /tmp/podman/##USER_NAME##/volumes
version:
  APIVersion: 4.9.4-dev
  Built: 1708537642
  BuiltTime: Thu Feb 22 01:47:22 2024
  GitCommit: ""
  GoVersion: go1.21.7 (Red Hat 1.21.7-1.module+el8.10.0+21318+5ea197f8)
  Os: linux
  OsArch: linux/amd64
  Version: 4.9.4-dev

Podman in a container

No

Privileged Or Rootless

Rootless

Upstream Latest Release

No

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[SoloGao]

Is there anyone can help looking at this problem? Thanks!

[rhatdan]

@giuseppe PTAL

[SoloGao]

Update:

1. I saw relevant tests in https://github.com/containers/podman/blob/6abf83fc42865f2f61c39dbbb150c7aafcbc80ba/test/e2e/pull_test.go#L68 . So it should have been tested. I tried the same config method(using --imagestore) but get no luck

2. I found a strange thing: The overlay.imagestore: /proj/podman/##USER_NAME##/images is my additionalimagestores.

Here is my storage.json.

[storage]
driver = "overlay"
runroot = "/run/user/##USER_UID##/containers"
graphroot = "/tmp/podman/##USER_NAME##"
imagestore = "/proj/podman/##USER_NAME##/imagestore"
[storage.options]
additionalimagestores = [
  "/proj/podman/##USER_NAME##/images",
]
pull_options = {enable_partial_images = "false", use_hard_links = "false", ostree_repos=""}
[storage.options.overlay]
mount_program = "/usr/bin/fuse-overlayfs"
mountopt = "nodev,metacopy=on"
force_mask="700"
[storage.options.thinpool]

3. When I remove additionalimagestores in storage.json, the overlay.imagestore entry in podman info disappears

[SoloGao]

Tried podman on rhel8.10 and 9.5 with the same imagestore/graphroot settings. Both root and rootless
8.10(podman 4.9.4): size increase in graphroot
9.5(podman 5.2.2): size increase in imagestore

[giuseppe]

Tried podman on rhel8.10 and 9.5 with the same imagestore/graphroot settings. Both root and rootless
8.10(podman 4.9.4): size increase in graphroot
9.5(podman 5.2.2): size increase in imagestore

are you saying it works well with podman 5.2.2?

[SoloGao]

Tried podman on rhel8.10 and 9.5 with the same imagestore/graphroot settings. Both root and rootless
8.10(podman 4.9.4): size increase in graphroot
9.5(podman 5.2.2): size increase in imagestore

are you saying it works well with podman 5.2.2?

Yes, podman 5.2.2 on rhel 9.5 works like a charm. Podman 4.9.4 on rhel8.10(afaik rhel8 will never get podman 5) not working.

[SoloGao]

The easiest method to reproduce:

1. Clean install RHEL8.10(even minimal)
2. yum install podman
3. vi /etc/containers/storage.conf, set imagestore = "/var/lib/containers/imagestore"
4. podman pull centos
5. cd /var/lib/containers
6. du -h --max-depth=1

0 ./sigstore
220M ./storage
748K ./imagestore
32K ./cache
220M .

[SoloGao]

Any ideas? @giuseppe sorry to bother you again~