cannot remove images on MacOS

[jianzhangbjz]

Issue Description

Cannot empty the space.

jiazha-mac:~ jiazha$ podman system prune -a
WARNING! This command removes:
	- all stopped containers
	- all networks not used by at least one container
	- all images without at least one container associated with them
	- all build cache

Are you sure you want to continue? [y/N] y
Error: 128 errors occurred:
	* open /var/home/core/.local/share/containers/storage/overlay-images/.tmp-images.json2632288465: no space left on device
	* open /var/home/core/.local/share/containers/storage/overlay-images/.tmp-images.json2123817899: no space left on device
	* open /var/home/core/.local/share/containers/storage/overlay-images/.tmp-images.json1495409303: no space left on device
	* open /var/home/core/.local/share/containers/storage/overlay-images/.tmp-images.json4215706309: no space left on device
	* ...

Steps to reproduce the issue

Steps to reproduce the issue
1.

jiazha-mac:~ jiazha$ podman build -f catalog.Dockerfile -t quay.io/olmqe/nginx-index:bug31645 .
Error: mkdir /var/tmp/libpod_builder3138920961: no space left on device

jiazha-mac:~ jiazha$ podman images 
REPOSITORY                                       TAG                                IMAGE ID      CREATED      SIZE
registry.redhat.io/redhat/redhat-operator-index  v4.16                              725ef7898c91  5 days ago   1.86 GB
<none>                                           <none>                             353830b4d40e  3 weeks ago  545 MB
<none>                                           <none>                             15457c8fe287  3 weeks ago  2.98 GB
quay.io/openshifttest/operator-framework-olm     cert5-rotation-rhel9               0dc376d40d53  3 weeks ago  1.94 kB
quay.io/openshifttest/operator-framework-olm     cert-rotation-rhel9                0dc376d40d53  3 weeks ago  1.94 kB
<none>                                           <none>                             d9a09b58e65a  3 weeks ago  524 MB
<none>                                           <none>                             933bb81719fc  3 weeks ago  3.07 GB
<none>                                           <none>                             18d5f115912d  3 weeks ago  545 MB
<none>                                           <none>                             6e3323203077  3 weeks ago  3.14 GB
quay.io/openshifttest/operator-framework-olm     cert6-rotation-rhel9               784cd33a21a5  3 weeks ago  515 MB
<none>                                           <none>                             e94c86fbd756  3 weeks ago  3.06 GB
<none>                                           <none>                             2cf2fc4591a8  3 weeks ago  822 MB
<none>                                           <none>                             ceed43eedfd9  3 weeks ago  4.74 GB
quay.io/openshifttest/operator-framework-olm     cert3-rotation-rhel9               a0f5554f7b1d  3 weeks ago  605 B
registry.ci.openshift.org/ocp/builder            rhel-9-golang-1.21-openshift-4.16  36b1ccb8e645  3 weeks ago  2.49 GB
registry.ci.openshift.org/ocp/4.16               base-rhel9                         45f7701f7772  3 weeks ago  400 MB
docker.io/library/golang                         1.21                               b76c4bf6405f  6 weeks ago  779 MB
registry.access.redhat.com/ubi9/ubi-minimal      latest                             49b88100b913  6 weeks ago  98.4 MB

jiazha-mac:~ jiazha$ podman rmi registry.redhat.io/redhat/redhat-operator-index:v4.16
Error: open /var/home/core/.local/share/containers/storage/overlay-images/.tmp-images.json428717241: no space left on device

Describe the results you received

There is no /var/home/ folder, I'm unsure why it tried to open it.

jiazha-mac:~ jiazha$ podman rmi registry.redhat.io/redhat/redhat-operator-index:v4.16
Error: open /var/home/core/.local/share/containers/storage/overlay-images/.tmp-images.json428717241: no space left on device
jiazha-mac:~ jiazha$ podman rmi 725ef7898c91
Error: 725ef7898c91: image not known

Describe the results you expected

I can remove those images successfully.

podman info output

If you are unable to run podman info for any reason, please provide the podman version, operating system and its version and the architecture you are running.

jiazha-mac:~ jiazha$ podman version
Client:       Podman Engine
Version:      5.0.0
API Version:  5.0.0
Go Version:   go1.22.1
Git Commit:   e71ec6f1d94d2d97fb3afe08aae0d8adaf8bddf0
Built:        Wed Mar 20 01:29:05 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.3
API Version:  5.0.3
Go Version:   go1.22.2
Built:        Fri May 10 08:00:00 2024
OS/Arch:      linux/arm64

jiazha-mac:~ jiazha$ podman info
host:
  arch: arm64
  buildahVersion: 1.35.4
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.85
    systemPercent: 0.04
    userPercent: 0.11
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.8-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 59138048
  memTotal: 2045304832
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.10.0-1.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.10.0
    package: netavark-1.10.3-3.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.10.3
  ociRuntime:
    name: crun
    package: crun-1.14.4-1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version 1.14.4
      commit: a220ca661ce078f2c37b38c92e66cf66c012d9c1
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240426.gd03c4e2-1.fc40.aarch64
    version: |
      pasta 0^20240426.gd03c4e2-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 531h 36m 24.00s (Approximately 22.12 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 106769104896
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 128
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.0.3
  Built: 1715299200
  BuiltTime: Fri May 10 08:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.2
  Os: linux
  OsArch: linux/arm64
  Version: 5.0.3
jiazha-mac:~ jiazha$ uname -a 
Darwin jiazha-mac 23.5.0 Darwin Kernel Version 23.5.0: Wed May  1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

[Luap99]

Duplicate of #13967

You best option is to podman machine rm and create a new one or manually delete some files in the VM until podman works again.

There is no /var/home/ folder, I'm unsure why it tried to open it.

Well all these paths are in the VM so you need podman machine ssh first.

[jianzhangbjz]

Hi @Luap99 , thanks for your updates! And, it still fails after recreting the machine, any suggest? Thanks!

jiazha-mac:~ jiazha$ podman machine stop
Machine "podman-machine-default" stopped successfully
jiazha-mac:~ jiazha$ podman machine rm 
The following files will be deleted:


/Users/jiazha/.config/containers/podman/machine/applehv/podman-machine-default.json
/var/folders/5n/w9ysf4w93jnfy7k19xxct31c0000gn/T/podman/podman-machine-default.sock
/var/folders/5n/w9ysf4w93jnfy7k19xxct31c0000gn/T/podman/podman-machine-default-gvproxy.sock
/var/folders/5n/w9ysf4w93jnfy7k19xxct31c0000gn/T/podman/podman-machine-default-api.sock
/var/folders/5n/w9ysf4w93jnfy7k19xxct31c0000gn/T/podman/podman-machine-default.log
Are you sure you want to continue? [y/N] y

jiazha-mac:~ jiazha$ podman machine init
Looking up Podman Machine image at quay.io/podman/machine-os:5.0 to create VM
Extracting compressed file: podman-machine-default-arm64.raw: done  
Machine init complete
To start your machine run:

	podman machine start

jiazha-mac:~ jiazha$ podman machine start
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful

API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully
jiazha-mac:~ jiazha$ podman machine list 
NAME                     VM TYPE     CREATED             LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  applehv     About a minute ago  Currently running  5           2GiB        100GiB

jiazha-mac:~ jiazha$ sudo podman build -f catalog.Dockerfile -t quay.io/olmqe/nginx-index:bug31645 .
Password:
Error: failed Request: Unable to copy tar file from request body /v5.0.0/libpod/build?dockerfile=%5B%22catalog.Dockerfile%22%5D&forcerm=1&httpproxy=1&identitylabel=1&idmappingoptions=%7B%22HostUIDMapping%22%3Atrue%2C%22HostGIDMapping%22%3Atrue%2C%22UIDMap%22%3A%5B%5D%2C%22GIDMap%22%3A%5B%5D%2C%22AutoUserNs%22%3Afalse%2C%22AutoUserNsOpts%22%3A%7B%22Size%22%3A0%2C%22InitialSize%22%3A0%2C%22PasswdFile%22%3A%22%22%2C%22GroupFile%22%3A%22%22%2C%22AdditionalUIDMappings%22%3Anull%2C%22AdditionalGIDMappings%22%3Anull%7D%7D&isolation=3&jobs=1&layers=1&networkmode=0&nsoptions=%5B%7B%22Name%22%3A%22user%22%2C%22Host%22%3Atrue%2C%22Path%22%3A%22%22%7D%5D&omithistory=0&output=quay.io%2Folmqe%2Fnginx-index%3Abug31645&outputformat=application%2Fvnd.oci.image.manifest.v1%2Bjson&pullpolicy=missing&rm=1&shmsize=67108864&t=quay.io%2Folmqe%2Fnginx-index%3Abug31645

[jianzhangbjz]

One more question, how can I specify the space? Thanks!

jiazha-mac:~ jiazha$ podman machine start
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful

Error: mkdir /var/lib/containers/storage/overlay/l: no space left on device
/usr/bin/bash: line 10: /etc/systemd/system.conf.d/default-env.conf: No space left on device
WARN[0009] API socket failed ping test                  
API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully

[Luap99]

Error: failed Request: Unable to copy tar file from request body /v5.0.0/libpod/build?dockerfile=%5B%22catalog.Dockerfile%22%5D&forcerm=1&httpproxy=1&identitylabel=1&idmappingoptions=%7B%22HostUIDMapping%22%3Atrue%2C%22HostGIDMapping%22%3Atrue%2C%22UIDMap%22%3A%5B%5D%2C%22GIDMap%22%3A%5B%5D%2C%22AutoUserNs%22%3Afalse%2C%22AutoUserNsOpts%22%3A%7B%22Size%22%3A0%2C%22InitialSize%22%3A0%2C%22PasswdFile%22%3A%22%22%2C%22GroupFile%22%3A%22%22%2C%22AdditionalUIDMappings%22%3Anull%2C%22AdditionalGIDMappings%22%3Anull%7D%7D&isolation=3&jobs=1&layers=1&networkmode=0&nsoptions=%5B%7B%22Name%22%3A%22user%22%2C%22Host%22%3Atrue%2C%22Path%22%3A%22%22%7D%5D&omithistory=0&output=quay.io%2Folmqe%2Fnginx-index%3Abug31645&outputformat=application%2Fvnd.oci.image.manifest.v1%2Bjson&pullpolicy=missing&rm=1&shmsize=67108864&t=quay.io%2Folmqe%2Fnginx-index%3Abug31645

How large is you build context directory?

The VM size is given with podman machine init --disk-size, I think 100 (GB) is the default

[jianzhangbjz]

100G as default.

jiazha-mac:~ jiazha$ podman build -f catalog.Dockerfile -t quay.io/olmqe/nginx-index:bug31645 .
ERRO[0241] 1 error occurred:
	* io: read/write on closed pipe

 
Error: failed Request: Unable to copy tar file from request body /v5.0.0/libpod/build?dockerfile=%5B%22catalog.Dockerfile%22%5D&forcerm=1&httpproxy=1&identitylabel=1&idmappingoptions=%7B%22HostUIDMapping%22%3Atrue%2C%22HostGIDMapping%22%3Atrue%2C%22UIDMap%22%3A%5B%5D%2C%22GIDMap%22%3A%5B%5D%2C%22AutoUserNs%22%3Afalse%2C%22AutoUserNsOpts%22%3A%7B%22Size%22%3A0%2C%22InitialSize%22%3A0%2C%22PasswdFile%22%3A%22%22%2C%22GroupFile%22%3A%22%22%2C%22AdditionalUIDMappings%22%3Anull%2C%22AdditionalGIDMappings%22%3Anull%7D%7D&isolation=3&jobs=1&layers=1&networkmode=0&nsoptions=%5B%7B%22Name%22%3A%22user%22%2C%22Host%22%3Atrue%2C%22Path%22%3A%22%22%7D%5D&omithistory=0&output=quay.io%2Folmqe%2Fnginx-index%3Abug31645&outputformat=application%2Fvnd.oci.image.manifest.v1%2Bjson&pullpolicy=missing&rm=1&shmsize=67108864&t=quay.io%2Folmqe%2Fnginx-index%3Abug31645
jiazha-mac:~ jiazha$ podman machine list 
NAME                     VM TYPE     CREATED      LAST UP            CPUS        MEMORY      DISK SIZE
podman-machine-default*  applehv     5 hours ago  Currently running  5           2GiB        100GiB
``