podman push image got Error: unexpected EOF #2123

fmhwong · 2024-08-09T19:27:59Z

Issue Description

When pushing an image to a remote repository
e.g. podman push icr.io/.....
the command will fail with Error: unexpected EOF
This worked in version 4.9.3

Steps to reproduce the issue

Build an image
Tag it with icr.io/....
podman push icr.io/...

Describe the results you received

The command failed with

Getting image source signatures
Copying blob sha256:38b40aba766b8cfd30c14ae2d26998668b33c771c9c7364684333d0de236d74a
Copying blob sha256:6aa7c734bdf18d74c9aacd71a0fca1c93346879de7b5f743bbb7e4979777b401
Copying blob sha256:2249eea5db0e73b91f7ae68667302a6f0503ccec1964bd87e7cc41f0cb14831d
Copying blob sha256:05afd36989e87599be7d14243f5c5f39c12607863c69479d88ef9ec5f33a6bb7
Copying blob sha256:5da3d903fc8c753ae3aad9a9277159ea9d4281027a5c26020cc700123fe97e6f
Copying blob sha256:3cbf0e69cdb6e1024ce229a3b7dbbc43742ed6551ecba2bfc00407a0e8238399
Copying blob sha256:65d2f740fb9eb8c2936ee855e6d64e10a97d2330518348be68648fb188208420
Copying blob sha256:710091c81ca585edefbe55e77af458016976d9dc1674b56d286ffc2584194e23
Copying blob sha256:2a1f794b41b7cd73739f40cd93def905be6912746d6ec2bf911b067f7546abc7
Error: unexpected EOF

Describe the results you expected

I expect the push would succeed. This fails on 5.2.0 but worked with 4.9.3.

podman info output

host:
  arch: arm64
  buildahVersion: 1.37.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 96.47
    systemPercent: 1.71
    userPercent: 1.82
  cpus: 6
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 501
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.9.11-200.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1634684928
  memTotal: 2044190720
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.12.0-1.20240801182221740927.main.8.g7cc82a4.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.13.0-dev
    package: netavark-1.12.1-1.20240802123603627403.main.4.geb4e5d3.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.13.0-dev
  ociRuntime:
    name: crun
    package: crun-1.15-1.20240731144341062436.main.62.ge6a1ef1.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ce56cc2473f934a77c79c5cc79a65fe157dec42d
      rundir: /run/user/501/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240624.g1ee2eca-1.fc40.aarch64
    version: |
      pasta 0^20240624.g1ee2eca-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/501/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.5
  swapFree: 0
  swapTotal: 0
  uptime: 0h 24m 43.00s
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 6631981056
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 39
  runRoot: /run/user/501/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.2.0
  Built: 1722556800
  BuiltTime: Thu Aug  1 20:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.5
  Os: linux
  OsArch: linux/arm64
  Version: 5.2.0

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

The text was updated successfully, but these errors were encountered:

sbrivio-rh · 2024-08-09T20:12:52Z

Does this always stop at the same point if you try again, that is:

Copying blob sha256:2a1f794b41b7cd73739f40cd93def905be6912746d6ec2bf911b067f7546abc7
Error: unexpected EOF

?

fmhwong · 2024-08-09T22:08:03Z

Yes

Vringe · 2024-08-12T11:29:13Z

Same here on 5.2.0

$ podman pull mariadb:10.6.18
$ podman tag mariadb:10.6.18 my-registry.internal/apps/mariadb:10.6.18
$ podman login my-registry.internal
Username: XXX
Password: XXX
Login Succeeded!
$ podman push my-registry.internal/apps/mariadb:10.6.18
Getting image source signatures
Copying blob sha256:f40b54b0983d5be2a6ce8b9f4734bf6f1a836460bbacc7fabc5af8690cbcb709
Copying blob sha256:3e0a5df725cad6ca4827cd96861e1d3c441d7f672cc47eae37efe68df300d6d5
Copying blob sha256:89ce5cb440de92f67498c06d1f5f62a5e2d2440b369aedc5b6105a5384a3754c
Copying blob sha256:048134f2a2bc84520c1cb976ce052a53408cc381f915aa896293cb7ef8e576ac
Copying blob sha256:a8c68591d421fc2d4bdda704f67a796edf5ff880c59358d75107eb5261821650
Copying blob sha256:605cd205119dc96c9d861bd5f5888e03ec674bb411df8519d997f3a68ceef339
Copying blob sha256:fdfa0472c150860607f1f9d2d1585e683eeea628cbfafd7d2c4b392d1085e5c0
Copying blob sha256:5576fb3fb8ef0baec62577e7e84d89499c35cc260ce2a2c19423749807b310ca
Error: unexpected EOF

Always failing at the same point.
Downgraded to 4.9.3 for now.

Luap99 · 2024-08-12T11:50:13Z

Are you both using podman machine?
In that case please run podman machine ssh to enter the VM and run the same push command there while adding --log-level debug and provide the full debug logs please.

Vringe · 2024-08-12T13:15:50Z

Looks like the memory footprint increased:

Copying blob 45f7ff791bb0 [--------------------------------------] 896.0KiB / 285.6MiB | 12.0 MiB/s
Copying blob 45f7ff791bb0 [=====>--------------------------------] 42.7MiB / 285.6MiB | 1.5 GiB/s
Copying blob 2bc5a73232de done   |
Copying blob 9845b4e6a5f2 [--------------------------------------] 0.0b / 18.0MiB | 0.0 b/s
Copying blob 956c99b07a41 [--------------------------------------] 0.0b / 65.0MiB | 0.0 b/s
Copying blob 7704d728820f done   |
Copying blob 78e126f97f4a done   |
Copying blob 45f7ff791bb0 [=====>--------------------------------] 47.1MiB / 285.6MiB | 1.6 GiB/s
Copying blob 2bc5a73232de done   |
Copying blob 9845b4e6a5f2 [==>-----------------------------------] 1.5MiB / 18.0MiB | 58.0 MiB/s
Copying blob 956c99b07a41 [--------------------------------------] 0.0b / 65.0MiB | 0.0 b/s
Copying blob 7704d728820f done   |
Copying blob 78e126f97f4a done   |
Copying blob 75d22ddbc2b6 [--------------------------------------] 0.0b / 12.5KiB | 0.0 b/s
Copying blob bd2dfaf7f59e [--------------------------------------] 0.0b / 27.0KiB | 0.0 b/s
DEBU[0003] exporting filesystem layer "beac8333c562321501684de93a41bd9153d12cc6bac898b162a9ddaeb83930db" without compression for blob "sha256:75d22ddbc2b655bfff67dd2623e5106520eb08ff0beada04dac6b7218e15774d"
DEBU[0003] No compression detected
DEBU[0003] Compressing blob on the fly
DEBU[0003] Uploading /v2/apps/mariadb/blobs/uploads/
DEBU[0003] POST https://my-registry.internal/v2/apps/mariadb/blobs/uploads/
DEBU[0003] exporting filesystem layer "18dfeb0946531a7a35548415b9e9322134452fde93ceb41240b14f2aff4e36af" without compression for blob "sha256:bd2dfaf7f59e81818105d456cd910aad3c7ed13911bcaf9054bd501791ca66c0"
DEBU[0003] No compression detected
DEBU[0003] Compressing blob on the fly
DEBU[0003] Uploading /v2/apps/mariadb/blobs/uploads/
DEBU[0003] POST https://my-registry.internal/v2/apps/mariadb/blobs/uploads/
DEBU[0003] No compression detected
DEBU[0003] Compressing blob on the fly
DEBU[0003] Uploading /v2/apps/mariadb/blobs/uploads/
DEBU[0003] POST https://my-registry.internal/v2/apps/mariadb/blobs/uploads/
DEBU[0003] PATCH https://my-registry.internal/v2/apps/mariadb/blobs/uploads/ab2c522c-df85-445b-9da8-ce3d82adb889?_state=XXX
Killed

I recreated the VM with 4GB of RAM (podman machine init -m 4096), which solved the issue for me.
The memory usage is now at around 2.5G during a push:

PID  USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2801 root      20   0 5328876   2.5g  34788 S   0.0  66.2   0:19.42 podman

fmhwong · 2024-08-12T13:26:49Z

DEBU[0003] No compression detected                      
DEBU[0003] Compressing blob on the fly                  
DEBU[0003] Uploading /v2/lumiere-dev/fmhwong/pipeline-tools/blobs/uploads/ 
DEBU[0003] POST https://icr.io/v2/lumiere-dev/fmhwong/pipeline-tools/blobs/uploads/ 
DEBU[0003] No compression detected                      
Copying blob 7a055536a569 [===>----------------------------------] 22.6MiB / 240.8MiB | 1.8 GiB/s
Copying blob 7812b0c7bd85 [--------------------------------------] 32.0KiB / 167.6MiB | 1.9 MiB/s
Copying blob 598ff0fc3f3a [>-------------------------------------] 5.8MiB / 392.7MiB | 83.6 MiB/s
Copying blob 7a055536a569 [===>----------------------------------] 22.9MiB / 240.8MiB | 1000.7 MiB/s
Copying blob 7812b0c7bd85 [--------------------------------------] 32.0KiB / 167.6MiB | 1.9 MiB/s
Copying blob 598ff0fc3f3a [>-------------------------------------] 5.8MiB / 392.7MiB | 83.6 MiB/s
Copying blob 5958ee998c8f [--------------------------------------] 32.0KiB / 10.7MiB | 3.2 MiB/s
Copying blob 1e1bdc7d850f [>-------------------------------------] 8.4MiB / 384.3MiB | 1.5 GiB/s
Copying blob 247c2d03e948 skipped: already exists  
Copying blob 7a055536a569 [===>----------------------------------] 23.9MiB / 240.8MiB | 1.2 GiB/s
Copying blob 7812b0c7bd85 [>-------------------------------------] 4.0MiB / 167.6MiB | 1.2 GiB/s
Copying blob 598ff0fc3f3a [>-------------------------------------] 6.2MiB / 392.7MiB | 150.5 MiB/s
Copying blob 7a055536a569 [===>----------------------------------] 24.5MiB / 240.8MiB | 1.4 GiB/s
Copying blob 7a055536a569 [======>-------------------------------] 42.4MiB / 240.8MiB | 935.4 MiB/s
Copying blob 7812b0c7bd85 [======>-------------------------------] 32.7MiB / 167.6MiB | 732.5 MiB/s
Copying blob 598ff0fc3f3a [>-------------------------------------] 13.3MiB / 392.7MiB | 667.6 MiB/s
Copying blob 7a055536a569 [======>-------------------------------] 43.3MiB / 240.8MiB | 1.2 GiB/s
Copying blob 7a055536a569 [==================>-------------------] 119.3MiB / 240.8MiB | 70.0 MiB/s
Copying blob 7812b0c7bd85 [==================================>---] 154.5MiB / 167.6MiB | 113.0 MiB/s
Copying blob 598ff0fc3f3a [====>---------------------------------] 53.3MiB / 392.7MiB | 93.9 MiB/s
Copying blob 5958ee998c8f done   | 
Copying blob 1e1bdc7d850f [====>---------------------------------] 46.8MiB / 384.3MiB | 119.9 MiB/s
Copying blob 247c2d03e948 skipped: already exists  
Copying blob 582d7370e5e5 done   | 
Killed

Luap99 · 2024-08-12T13:30:28Z

@mtrmac PTAL

fmhwong · 2024-08-12T16:47:20Z

Thanks @Vringe for the workaround. I set the memory to 6GB and that solved the problem.

mtrmac · 2024-08-12T20:19:36Z

Is containers.conf set to use a zstd:chunked compression?

Compare containers/storage#2055 .

Vringe · 2024-08-13T07:00:27Z

Hmm, nope I checked .config/containers/containers.conf on my local machine and inside the machine. There is no compression settings set.

I actually just installed podman yesterday from the standard homebrew formula without changing anything at all.

Luap99 · 2024-08-13T11:18:34Z

In the machine VM.

$ grep compression_format /usr/share/containers/containers.conf 
#compression_format = "gzip"
compression_format = "zstd:chunked"
$ rpm -q containers-common
containers-common-0.60.0-1.20240731140954782684.main.10.gae4a61e1.fc40.noarch

I am not sure why we still ship the wrong config files in the podman-next copr. My assumption was that this should only be set on rawhide/f41.
cc @lsm5 @baude

rhatdan · 2024-08-13T12:49:34Z

I agree.

zstd:chunked should follow distro policies and no blanket handling for all copr envs. Resolves: containers#2123 Signed-off-by: Lokesh Mandvekar <[email protected]>

Given we ship podman-next copr packages on quite a few envs now, config file handling should follow distro policy as much as possible. Resolves: containers#2123 Signed-off-by: Lokesh Mandvekar <[email protected]>

thoangtrvn · 2024-11-05T15:09:34Z

From my side, the issue is resolved after upgrading Podman to 5.2.5. I have to delete the VM, recreate it so that it uses the new Podman. Meaning all the existing images have to be re-created, and that will solve the problem.

fmhwong added the kind/bug label Aug 9, 2024

lsm5 transferred this issue from containers/podman Aug 13, 2024

lsm5 added a commit to lsm5/common that referenced this issue Aug 13, 2024

[skip-ci] RPM: No zstd:chunked for copr envs

6794b88

zstd:chunked should follow distro policies and no blanket handling for all copr envs. Resolves: containers#2123 Signed-off-by: Lokesh Mandvekar <[email protected]>

lsm5 mentioned this issue Aug 13, 2024

RPM: No blanket conditionals for configs on COPR envs #2122

Merged

openshift-merge-bot bot closed this as completed in #2122 Aug 14, 2024

openshift-merge-bot bot closed this as completed in 2e812ea Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

podman push image got Error: unexpected EOF #2123

podman push image got Error: unexpected EOF #2123

fmhwong commented Aug 9, 2024

sbrivio-rh commented Aug 9, 2024

fmhwong commented Aug 9, 2024

Vringe commented Aug 12, 2024 •

edited

Loading

Luap99 commented Aug 12, 2024

Vringe commented Aug 12, 2024

fmhwong commented Aug 12, 2024

Luap99 commented Aug 12, 2024

fmhwong commented Aug 12, 2024

mtrmac commented Aug 12, 2024

Vringe commented Aug 13, 2024

Luap99 commented Aug 13, 2024

rhatdan commented Aug 13, 2024

thoangtrvn commented Nov 5, 2024

podman push image got Error: unexpected EOF #2123

podman push image got Error: unexpected EOF #2123

Comments

fmhwong commented Aug 9, 2024

Issue Description

Steps to reproduce the issue

Describe the results you received

Describe the results you expected

podman info output

Podman in a container

Privileged Or Rootless

Upstream Latest Release

Additional environment details

Additional information

sbrivio-rh commented Aug 9, 2024

fmhwong commented Aug 9, 2024

Vringe commented Aug 12, 2024 • edited Loading

Luap99 commented Aug 12, 2024

Vringe commented Aug 12, 2024

fmhwong commented Aug 12, 2024

Luap99 commented Aug 12, 2024

fmhwong commented Aug 12, 2024

mtrmac commented Aug 12, 2024

Vringe commented Aug 13, 2024

Luap99 commented Aug 13, 2024

rhatdan commented Aug 13, 2024

thoangtrvn commented Nov 5, 2024

Vringe commented Aug 12, 2024 •

edited

Loading