cAdvisor Fails to Retrieve Metrics for Podman Container Managed by systemd

[dmennecke1]

Issue Description

When running an NGINX container manually using the following command:

podman run --name=nginx -e=CONTAINER_NAME=nginx -p=8081:80 --restart=always -d docker.io/nginx

cAdvisor is able to successfully retrieve and display container metrics:

container_cpu_load_average_10s{container_label_BACKEND_URL="",container_label_ERROR_FILE="",container_label_io_buildah_version="",container_label_maintainer="NGINX Docker Maintainers <docker [email protected]>",container_label_org_opencontainers_image_ref_name="",container_label_org_opencontainers_image_version="",id="/user.slice/user-1001.slice/[email protected]/user.slice/libpod-76e7df60cd24ecdabca6f5ef54b2ba87bbca060adeb0018ddb2c0017f18c50db.scope",image="docker.io/library/nginx:latest",name="nginx"} 0 1737716804296

However, when starting the container as a systemd service (based on the Quadlet feature), cAdvisor fails to retrieve metrics.

Systemd Service Configuration

The container is started using the following systemd service unit:

# Automatically generated by /usr/local/lib/systemd/user-generators/podman-user-generator
#
[X-Container]
ContainerName=nginx
Image=docker.io/nginx
PublishPort=8081:80

[Install]
WantedBy=default.target

[Unit]
Wants=podman-user-wait-network-online.service
After=podman-user-wait-network-online.service
SourcePath=/etc/containers/systemd/users/1001/nginx.container
RequiresMountsFor=%t/containers

[Service]
Environment=PODMAN_SYSTEMD_UNIT=%n
KillMode=mixed
ExecStop=/usr/local/bin/podman rm -v -f -i --cidfile=%t/%N.cid
ExecStopPost=-/usr/local/bin/podman rm -v -f -i --cidfile=%t/%N.cid
Delegate=yes
Type=notify
NotifyAccess=all
SyslogIdentifier=%N
ExecStart=/usr/local/bin/podman run --name nginx --cidfile=%t/%N.cid --replace --rm --cgroups=split --sdnotify=conmon -d --publish 8081:80 --env CONTAINER_NAME=nginx  docker.io/nginx

Observed Errors

When the container is started via systemd, the following cAdvisor errors appear in the logs:

E0124 12:16:27.057400       1 manager.go:1116] Failed to create existing container: /user.slice/user-1001.slice/[email protected]/app.slice/nginx.service/libpod-payload-9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d: unable to determine 9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d rw layer id

W0124 12:16:27.109060       1 manager.go:1169] Failed to process watch event {EventType:0 Name:/user.slice/user-1001.slice/[email protected]/app.slice/nginx.service/libpod-payload-9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d WatchSource:0}: unable to determine 9aeda1a5000c897b2e1b294211bffe2d519f9dffc1fbb6c575f215bd8dc9076d rw layer id

Expected Behavior

cAdvisor should return metrics for the container when it is managed by systemd, similar to the manual Podman execution.

Actual Behavior

cAdvisor fails to collect container metrics and logs errors related to RW layer ID determination.