Modifying an existing file next to a cache mount resets mode&ownership of parent directories #27777
Description
Issue Description
Hi, this is a pretty strange bug, but here we are. When using a RUN --mount=type=cache while touching a previously existing directory (or a file within) next to the cache mount, the whole chain of parent directories (from the cache mount parent up to /) get botched mode&ownership flags (seems like a reset).
This was very hard to pinpoint, touching a file in the same directory hides the problem, operating on/in the cache mount doesn't trigger it.
Steps to reproduce the issue
Here's a rather minimalist but verbose Dockerfile to reproduce the issue:
FROM docker.io/library/ubuntu:latest
ARG WORKDIR=/home/testuser
ARG UID=1234
RUN set -eux ;\
mkdir -p "$WORKDIR" ;\
useradd -s /usr/sbin/nologin -d "$WORKDIR" -M -g 0 -u "$UID" testuser ;\
chown -R "$UID:0" "$WORKDIR"
USER $UID
WORKDIR $WORKDIR
RUN \
# let's create a directory owned by the user, notice the . directory (/home/testuser) is owned by it too
mkdir -p somedir ;\
ls -al ;\
:
# let's mount a cache somewhere inside the workdir
RUN \
--mount=type=cache,target=.cache \
# alone it's not enough to cause something, but as soon an existing directory is touched
# the whole parent tree of the workdir ownership/mode gets messed up:
# thiss triggers it
#touch somedir/newfile ;\
# this too
touch somedir ;\
# this doesn't, though, in fact it partially prevents the problem
#touch newfile ;\
# still at this moment there's nothing wrong
ls -al ;\
:
#until now
RUN \
ls -dla /;\
ls -dla /home ;\
ls -al ;\
:Build it with:
podman build -t workdir-test:local --no-cache .- or (for reference)
docker build -t workdir-test:local --no-cache --progress=plain .
Describe the results you received
Run the container, checking the flags:
- podman
$ podman run --rm -it -u0 workdir-test:local sh -c 'ls -dal /; ls -dal /home; ls -al /home/testuser;' dr-xr-xr-x. 1 root root 12 Dec 15 23:36 / <<< 0555 instead of 0755, but that's most likely unrelated drwxr-xr-x. 1 root root 16 Dec 15 21:58 /home total 0 drwxr-xr-x. 1 root root 14 Dec 15 21:58 . <<< uid(root) instead of uid(testuser), this is it drwxr-xr-x. 1 root root 16 Dec 15 21:58 .. drwxr-xr-x. 1 testuser root 0 Dec 15 21:58 somedir - docker
$ docker run --rm -it -u0 workdir-test:local sh -c 'ls -dal /; ls -dal /home; ls -al /home/testuser;' drwxr-xr-x. 1 root root 0 Dec 15 23:36 / drwxr-xr-x. 1 root root 16 Dec 15 21:35 /home total 12 drwxr-xr-x. 1 testuser root 14 Dec 15 21:35 . <<< uid(testuser), no change drwxr-xr-x. 1 root root 16 Dec 15 21:35 .. drwxr-xr-x. 1 testuser root 0 Dec 15 21:35 somedir
wagoodman/dive also produces quite some weird output:
- podman (
dive <(podman save localhost/workdir-test:local) --source=docker-archive, 3rd layer)┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━ Permission UID:GID Size Filetree ---------- 0:0 4.8 kB ├── home ---------- 0:0 0 B │ ├── testuser drwxr-xr-x 1234:0 0 B │ │ └── somedir - docker (
dive <(docker save workdir-test:local) --source=docker-archive)┃ ● Current Layer Contents ┣━━━━━━━━━━━━━━━━━━━━━━━━━━ Permission UID:GID Size Filetree drwxr-xr-x 0:0 4.8 kB ├── home drwxr-xr-x 1234:0 0 B │ ├── testuser drwxr-xr-x 1234:0 0 B │ │ └── somedir
Describe the results you expected
I'd definitely expect such an innocent command - run under a local user - wouldn't reset the owner/permissions of the parent directory.
podman info output
host:
arch: amd64
buildahVersion: 1.42.2
cgroupControllers:
- cpu
- io
- memory
- pids
cgroupManager: systemd
cgroupVersion: v2
conmon:
package: conmon-2.1.13-1.fc42.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.1.13, commit: '
cpuUtilization:
idlePercent: 89.93
systemPercent: 3.48
userPercent: 6.59
cpus: 8
databaseBackend: sqlite
distribution:
distribution: fedora
variant: workstation
version: "42"
emulatedArchitectures:
- linux/arm
- linux/arm64
- linux/arm64be
- linux/loong64
- linux/mips
- linux/mips64
- linux/ppc
- linux/ppc64
- linux/ppc64le
- linux/riscv32
- linux/riscv64
- linux/s390x
eventLogger: journald
freeLocks: 2048
hostname: feathora
idMappings:
gidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 6.17.12-200.fc42.x86_64
linkmode: dynamic
logDriver: journald
memFree: 23535853568
memTotal: 33279598592
networkBackend: netavark
networkBackendInfo:
backend: netavark
dns:
package: aardvark-dns-1.17.0-1.fc42.x86_64
path: /usr/libexec/podman/aardvark-dns
version: aardvark-dns 1.17.0
package: netavark-1.17.1-1.fc42.x86_64
path: /usr/libexec/podman/netavark
version: netavark 1.17.1
ociRuntime:
name: crun
package: crun-1.25.1-1.fc42.x86_64
path: /usr/bin/crun
version: |-
crun version 1.25.1
commit: 156ae065d4a322d149c7307034f98d9637aa92a2
rundir: /run/user/1000/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
os: linux
pasta:
executable: /usr/bin/pasta
package: passt-0^20250919.g623dbf6-1.fc42.x86_64
version: |
pasta 0^20250919.g623dbf6-1.fc42.x86_64
Copyright Red Hat
GNU General Public License, version 2 or later
<https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
remoteSocket:
exists: true
path: /run/user/1000/podman/podman.sock
rootlessNetworkCmd: pasta
security:
apparmorEnabled: false
capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
rootless: true
seccompEnabled: true
seccompProfilePath: /usr/share/containers/seccomp.json
selinuxEnabled: true
serviceIsRemote: false
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.3.1-2.fc42.x86_64
version: |-
slirp4netns version 1.3.1
commit: e5e368c4f5db6ae75c2fce786e31eef9da6bf236
libslirp: 4.8.0
SLIRP_CONFIG_VERSION_MAX: 5
libseccomp: 2.5.5
swapFree: 8589930496
swapTotal: 8589930496
uptime: 0h 5m 37.00s
variant: ""
plugins:
authorization: null
log:
- k8s-file
- none
- passthrough
- journald
network:
- bridge
- macvlan
- ipvlan
volume:
- local
registries:
localhost:5000:
Blocked: false
Insecure: true
Location: localhost:5000
MirrorByDigestOnly: false
Mirrors: null
Prefix: localhost:5000
PullFromMirror: ""
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- docker.io
store:
configFile: /home/tlwhitec/.config/containers/storage.conf
containerStore:
number: 0
paused: 0
running: 0
stopped: 0
graphDriverName: overlay
graphOptions: {}
graphRoot: /home/tlwhitec/.local/share/containers/storage
graphRootAllocated: 427133239296
graphRootUsed: 376377151488
graphStatus:
Backing Filesystem: btrfs
Native Overlay Diff: "true"
Supports d_type: "true"
Supports shifting: "false"
Supports volatile: "true"
Using metacopy: "false"
imageCopyTmpDir: /var/tmp
imageStore:
number: 56
runRoot: /run/user/1000/containers
transientStore: false
volumePath: /home/tlwhitec/.local/share/containers/storage/volumes
version:
APIVersion: 5.7.1
BuildOrigin: Fedora Project
Built: 1765324800
BuiltTime: Wed Dec 10 01:00:00 2025
GitCommit: f845d14e941889ba4c071f35233d09b29d363c75
GoVersion: go1.24.10
Os: linux
OsArch: linux/amd64
Version: 5.7.1Podman in a container
No
Privileged Or Rootless
Rootless
Upstream Latest Release
Yes
Additional environment details
Just a regular Fedora 42 workstation.
Additional information
I was primarily testing this as rootless, but I hit the same bug in rootful too.
I can't recreate the problem in my Ubuntu 24.04 server with it's podman 4.9.3.