WIP: Add a `/etc/containers/auth.json` #1746

cgwalters · 2022-12-08T16:28:39Z

A long-running tension in the docker/podman land is around running as a system service versus being executed by a user. (Specifically a "login user", i.e. a Unix user that can be logged
into via ssh etc.)

For login users, it makes total sense to configure the container
runtime in $HOME.

But for system services (e.g. code executed by systemd) it
is generally a bad idea to access or read the /root home
directory. On image based systems, /root may be dynamically
mutable state in contrast to /etc which may be managed
by OS upgrades, or even be read-only.

For these reasons, let's introduce /etc/contaners/auth.json.
If it is present, and the current process is executing in
systemd, it will be preferred. (There's some further logic
around this that is explained in the manpage; please see that
for details)

cc coreos/rpm-ostree#4180

Signed-off-by: Colin Walters [email protected]

cgwalters · 2022-12-08T16:30:23Z

Specifically for example I plan to ensure that bootc to reads this file and not /root/.docker etc by default. For podman to ensure maximum compatibility I think we need to add this file into the set.

cgwalters · 2022-12-08T16:31:14Z

(Lightly tested with an override to point skopeo at this)

cgwalters · 2022-12-08T16:34:43Z

For OpenShift I think it would make sense to have /var/lib/kubelet/config.json be a symlink to /etc/containers/auth.json. This would fix a pile of long-running hacks that hardcode podman run --authfile=/var/lib/kubelet/config.json.

mtrmac

Thanks! Using a PR to discuss the details is definitely helpful. Just a quick first impression for now.

For OpenShift I think it would make sense to have /var/lib/kubelet/config.json be a symlink to /etc/containers/auth.json.

Yes, if there is any kind of computer where having a system-wide credentials file makes sense, an OpenShift node should be that. OTOH does this mean we are giving up on some SELinux restrictions on root? The /etc/containers path is necessarily unrestricted. I didn’t look into how the Kubelet is, or isn’t restricted.

Maybe we could do it the other way, and symlink /etc/containers/auth.json to the Kubelet path.

pkg/docker/config/config.go

mtrmac · 2022-12-08T18:08:31Z

pkg/docker/config/config.go

@@ -204,7 +208,17 @@ func GetAllCredentials(sys *types.SystemContext) (map[string]types.DockerAuthCon
 // The homeDir parameter should always be homedir.Get(), and is only intended to be overridden
 // by tests.
 func getAuthFilePaths(sys *types.SystemContext, homeDir string) []authPath {


Adding the new paths to this function means that writers would never write to systemPath without a specific override; is that intended?

I think that’s actually a good idea, so that root’s interactive login doesn’t write to a noninteractively-used file, but it should be documented.

mtrmac · 2022-12-08T18:10:52Z

pkg/docker/config/config.go

@@ -204,7 +208,17 @@ func GetAllCredentials(sys *types.SystemContext) (map[string]types.DockerAuthCon
 // The homeDir parameter should always be homedir.Get(), and is only intended to be overridden
 // by tests.
 func getAuthFilePaths(sys *types.SystemContext, homeDir string) []authPath {
+	runningInSystemd := os.Getenv("INVOCATION_ID") != ""
+	runningAsRoot := os.Getuid() == 0


At a first glance it seems that this should use rootless.GetRootlessEUID. (Admittedly the rest of this file doesn‘t, and it probably should as well…)

pkg/docker/config/config.go

mtrmac · 2022-12-08T18:19:08Z

pkg/docker/config/config.go

+		}
+		perms := st.Mode().Perm()
+		if (perms & 04) > 0 {
+			return dockerConfigFile{}, fmt.Errorf("refusing to process %s with world read permissions", path)


The documentation of the field says “readable by group or other”, the man page says “mode 0600”.

I’m not sure what we want[1] but it should be consistent.

[1] Starting with 0600 would allow us to possibly relax the restriction later, so that seems to be the conservative choice. Also, we are only reading the file when the user is root, so no other permissions seem necessary.

For now I chose to error on group/world read permissions. Forcing 0600 also forces write permissions, but I can imagine cases where one might not want that. (Not that it really matters because root usually has CAP_DAC_OVERRIDE)

mtrmac · 2022-12-08T18:20:17Z

pkg/docker/config/config.go

@@ -204,7 +208,17 @@ func GetAllCredentials(sys *types.SystemContext) (map[string]types.DockerAuthCon
 // The homeDir parameter should always be homedir.Get(), and is only intended to be overridden
 // by tests.
 func getAuthFilePaths(sys *types.SystemContext, homeDir string) []authPath {


With growing complexity, it seems more important to have unit tests for this. But that’s the very last step.

cgwalters · 2022-12-08T18:53:32Z

OTOH does this mean we are giving up on some SELinux restrictions on root? The /etc/containers path is necessarily unrestricted. I didn’t look into how the Kubelet is, or isn’t restricted.

Good question. Today,

$ ll -Z /etc/containers/
total 12
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0        0 Oct  6 15:46 certs.d
drwx------. 1 root root unconfined_u:object_r:etc_t:s0   26 Oct 30 17:55 networks
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0       14 Oct  6 15:46 oci
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      569 Oct  6 15:46 policy.json
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0     3897 Oct  6 15:46 registries.conf
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0       38 Oct  6 15:46 registries.conf.d
drwxr-xr-x. 1 root root system_u:object_r:etc_t:s0      132 Nov 17 19:53 registries.d
-rw-r--r--. 1 root root system_u:object_r:etc_t:s0      699 Dec  9  2021 toolbox.conf

It could make sense to introduce a dedicated container_auth_t or so. But we can do that even inside /etc/containers. Or, we could have it be in /etc/containers/secrets and label the directory for future expansion - and if we do that it'd make sense to have the dir also mode 0700 for example for stock DAC protections.

cgwalters · 2022-12-08T19:07:03Z

A variant of this is that we could:

Just add a change to the man page here which documents the possible existence of /etc/containers/auth.json
I change bootc to use it by default

This I think would get about 45% of the value with 1% of the work.

mtrmac · 2022-12-08T19:07:44Z

I’m not thinking so much whether we could isolate this in root with SELinux in principle, it’s more that we can’t do that effectively: once /etc/containers/…auth.json becomes the Podman default for root, all plausible SELinux contexts that could run Podman must be allowed to access that file. The only way to truly isolate is to have more isolated files, i.e. not the system-wide /etc/containers/*.

But that’s just me rehashing the previous discussion, I guess. It seems to me that single-purpose appliances can use /etc/containers, traditional multi-user multi-purpose UNIX servers shouldn’t set it up at all, and should continue to use whatever they are doing now.

mtrmac · 2022-12-08T19:10:15Z

A variant of this is that we could: …

Overall, so far I’m not convinced that adding this default path is clearly worthwhile. But it’s useful enough in some contexts that I’m fine with it being added.

cgwalters · 2022-12-08T19:20:11Z

I’m not thinking so much whether we could isolate this in root with SELinux in principle, it’s more that we can’t do that effectively: once /etc/containers/…auth.json becomes the Podman default for root, all plausible SELinux contexts that could run Podman must be allowed to access that file.

I think it is possible because today most invocations of podman are going to start from unconfined_t or init_t, and we'd be sure that those contexts are allowed to access the file. The value of the distinct context is that lesser-privileged domains (say NetworkManager_t) would not be able to read it by default.

But that’s just me rehashing the previous discussion, I guess. It seems to me that single-purpose appliances can use /etc/containers, traditional multi-user multi-purpose UNIX servers shouldn’t set it up at all, and should continue to use whatever they are doing now.

Well, we have to account for mixed setups. For example, a valid user story here is a company/organization deploying Linux desktop systems using bootc and having that system upgrade use one pull secret, which would be distinct from the secret they use (today) in $HOME or /root for their application containers.

So I see this as just adding a dedicated file for "the system" without conflicting with any optional setup.

Overall, so far I’m not convinced that adding this default path is clearly worthwhile. But it’s useful enough in some contexts that I’m fine with it being added.

Thanks; this is a useful debate to have for sure because once added, it can't be removed and so there's long term maintenance implications for sure.

mtrmac · 2022-12-09T20:47:50Z

I think it is possible because today most invocations of podman are going to start from unconfined_t or init_t, and we'd be sure that those contexts are allowed to access the file. The value of the distinct context is that lesser-privileged domains (say NetworkManager_t) would not be able to read it by default.

(BTW OpenShift, at least in some versions, runs Podman from inside a NetworkManager hook… it’s containers all the way down.)

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746

cgwalters · 2022-12-15T22:31:18Z

At this point it seems useful to make readJSONFile, findCredentialsInFile (and other functions?) methods on authPath, instead of passing the individual parameters; ideally in a separate commit.

Done in #1765
(unit tests don't quite pass for me locally, but they fail in the same way without this. They seem to assume some global state)

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746 Signed-off-by: Colin Walters <[email protected]>

mtrmac · 2022-12-16T19:38:55Z

(unit tests don't quite pass for me locally, but they fail in the same way without this. They seem to assume some global state)

I wouldn’t be too surprised if they were insufficiently isolated from the per-user state. If so, that’s not intentional.

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746 Signed-off-by: Colin Walters <[email protected]>

A long-running tension in the docker/podman land is around running as a system service versus being executed by a user. (Specifically a "login user", i.e. a Unix user that can be logged into via `ssh` etc.) For login users, it makes total sense to configure the container runtime in `$HOME`. But for system services (e.g. code executed by systemd) it is generally a bad idea to access or read the `/root` home directory. On image based systems, `/root` may be dynamically mutable state in contrast to `/etc` which may be managed by OS upgrades, or even be read-only. For these reasons, let's introduce `/etc/contaners/auth.json`. If it is present, and the current process is executing in systemd, it will be preferred. (There's some further logic around this that is explained in the manpage; please see that for details) cc coreos/rpm-ostree#4180 Signed-off-by: Colin Walters <[email protected]>

cgwalters · 2023-01-03T15:15:45Z

(BTW OpenShift, at least in some versions, runs Podman from inside a NetworkManager hook… it’s containers all the way down.)

Yeah true 😢

But an attempt to default SELinux confine NetworkManager hooks IIRC failed, and I think the status quo there is the default is unconfined.

cgwalters · 2023-01-03T15:16:20Z

OK I rebased 🏄 this - it's definitely cleaner on top of #1765

Fixed some comments and issues, but still more to do.

mtrmac · 2023-01-04T00:27:10Z

pkg/docker/config/config.go

+			return dockerConfigFile{}, fmt.Errorf("stat %s: %w", path.path, err)
+		}
+		perms := st.Mode().Perm()
+		if (perms & 044) > 0 {


!= 0 seems a better fit with the “bit set” format.

pkg/docker/config/config.go

cgwalters · 2023-02-17T15:13:06Z

We might also want to investigate https://systemd.io/CREDENTIALS/ here.

cgwalters · 2023-10-17T18:12:29Z

We might also want to investigate https://systemd.io/CREDENTIALS/ here.

OK yes I dug into this a bit more recently and I think it'd be nice to support this more natively. The credentials path is a bit orthogonal in a way to a plain old /etc/containers/auth.json, but it has a lot of nicer properties.

My proposal here is basically we document and support a flow like this:

systemd-creds encrypt --name='' - - < /path/to/docker.cfg > /etc/containers/system-auth.json

This is invoked by provisioning/configuration management tooling (and the choice of filename here is just a recommendation). Same as all other uses of systemd credentials.

Then anything which wants to read this (e.g. skopeo inspect/podman pull) needs to be invoked from a systemd unit, and set LoadCredentialEncrypted=containers-auth.json:/etc/containers/system-auth.json.

Now what we would do inside containers/image here is check whether $CREDENTIALS_DIRECTORY/containers-auth.json exists - if it does it is preferred over anything else.

Here's an example command I used to test this flow:

$ systemd-run -p LoadCredentialEncrypted=containers-auth.json:/etc/containers/system-auth.json -qPG /bin/sh -c 'podman pull --authfile $CREDENTIALS_DIRECTORY/containers-auth.json docker://registry.ci.openshift.org/ocp/release:4.15.0-0.nightly-2023-10-17-065657'

Except the benefit of supporting $CREDENTIALS_DIRECTORY/containers-auth.json automatically is exactly that it becomes very ergonomic to use! We encourage people to use credentials for this. Also it becomes easier to juggle multiple pull secrets because one can do e.g.:

systemd-creds encrypt --name="" - - < /path/to/other-docker.cfg > /etc/containers/system-auth-internalregistry.json

(Cleanly separating out a distinct pull secret)

Then when invoking systemd units, one would set LoadCredentialEncrypted=containers-auth.json:/etc/containers/system-auth-internalregistry.json and if we did the automatic binding, that would mean the "internalregistry" pull secret is automatically used for all invocations of podman/skopeo etc.

cgwalters · 2024-03-27T11:03:39Z

I recently came across https://access.redhat.com/solutions/7002142 which has a really bad suggestion for how to make podman login persist. I do think this PR still makes sense; will try to rebase at some point soon and pick this back up.

vrothberg · 2024-06-26T10:02:36Z

@cgwalters, what are your thoughts on this issue today?

I am carefully walking over the bootc docs and think that the status quo does not offer a good UX. Having an /etc/containers/auth.json seems like an intuitive way forward but maybe some conversations have changed direction in the meantime.

I didn't read the code yet but would expect that /etc/containers/auth.json is only used for reading credentials?

rhatdan · 2024-06-26T10:50:30Z

What would the protections on that file be by default 600 would be my suggestion, and then people who wanted to share it with non root users could open it up, since it has a non encrypted password embedded in it.

cgwalters · 2024-06-26T12:00:30Z

@cgwalters, what are your thoughts on this issue today?

Oh yes I think we should definitely still do this.

mtrmac · 2024-06-26T17:25:14Z

I am carefully walking over the bootc docs and think that the status quo does not offer a good UX.

@vrothberg That link isn’t publicly available

mtrmac · 2024-06-26T17:34:36Z

My thoughts in general:

I don’t really see value in a systemd-service-specific feature, the way this PR works now. A systemd service can be reasonably (and with things like Quadlet, quite easily and in an automated way!) set up to have its own service-specific secret file, and to pass --authfile to only use those credentials instead of whatever else is in the environment. Making it the default workflow to use an explicit --authfile is attractive to me.
I don’t feel strongly about a system-wide /etc/containers/auth.json read by everything. There are systems where “everything has access to a credential” can be valuable, and certainly easier than manually managing credentials in all relevant contexts. OTOH it needs to be constrained to such specific single-purpose systems, it should never be the default way flexible systems are built. (In particular, ISV’s software installation instructions for generally-applicable software should never tell users to put credentials into a system-readable file.) So… this would make work a tiny bit easier for users who are already doing a lot of work to build a single-purpose appliance anyway.
We’ve had a conversation about RHEL subscription management automatically setting up registry credentials. Now for that, we do need a system-wide config file — but it also probably should not be a single file, but a *.d directory, so that it can be managed per-repo (per product subscription) without conflicts. Or, alternatively, we would have to figure out local locking (Multiple simultaneous podman login commands can lose entries #1365 ), and ensure that everything follows the rules — supporting *.d seems easier.

cgwalters · 2024-06-26T20:06:50Z

I'd agree that in the general case of per-image/per-service pull secrets makes total sense.

But I do feel for people trying out container-based systems at a smaller scale (especially the important proof-of-concept phase), it's highly likely they start with just one authenticated registry and they'd be perfectly fine with a global secret to start (especially one that applied to bootc and podman), with the ability to override for individual images being the next step.

but it also probably should not be a single file, but a *.d directory, so that it can be managed per-repo (per product subscription) without conflicts.

This would definitely be nice indeed. For clarity and completeness, this would hopefully really be having an auth file stack that implemented the https://uapi-group.org/specifications/specs/configuration_files_specification/

mtrmac · 2024-06-27T14:34:54Z

For clarity and completeness, this would hopefully really be having an auth file stack that implemented the https://uapi-group.org/specifications/specs/configuration_files_specification/

As long as they are at the end of the search list = do not incur a cost for applications with more explicit credential locations, why not.

cgwalters · 2024-10-01T20:19:36Z

I randomly came across https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/8/html-single/composing_a_customized_rhel_system_image/index#con_the-container-registry-credentials_assembly_pushing-a-container-to-a-register-and-embedding-it-into-a-image today which created another path for pull secrets - if I had to guess at the rationale it'd be similar.

mtrmac · 2024-10-01T20:44:16Z

That looks like a single application having a place to configure credentials used for that application (and, of course, it’s fine for such things to exist); if that expresses a need for need for a system-wide credential store, I don’t know enough to see it.

cgwalters · 2024-10-01T21:10:47Z

I do continue to think that many simple use cases would be happy with a default global path and
if we had a global .d directory, we would have at least referenced it from that doc as an option versus an app-specific path.

cgwalters · 2024-10-23T15:57:47Z

Closing in favor of #2600 where development continues

cgwalters mentioned this pull request Dec 8, 2022

Document how to authenticate to private container repository. coreos/rpm-ostree#4180

Closed

cgwalters requested a review from mtrmac December 8, 2022 16:30

mtrmac reviewed Dec 8, 2022

View reviewed changes

mtrmac added the kind/feature A request for, or a PR adding, new functionality label Dec 8, 2022

cgwalters added a commit to cgwalters/image that referenced this pull request Dec 15, 2022

config: use authPath struct consistently

79ea7c2

Instead of packing and unpacking it into two variables in various places. Prep for adding a third member of this struct as part of containers#1746

cgwalters mentioned this pull request Dec 15, 2022

config: use authPath struct consistently #1765

Merged

cgwalters force-pushed the add-system-path branch 2 times, most recently from 76b414f to 838b0f9 Compare January 3, 2023 15:08

cgwalters force-pushed the add-system-path branch from 838b0f9 to e998b2f Compare January 3, 2023 15:10

mtrmac reviewed Jan 4, 2023

View reviewed changes

cgwalters mentioned this pull request Sep 19, 2023

man: Add ostree-container-auth ostreedev/ostree-rs-ext#543

Merged

cgwalters mentioned this pull request Oct 17, 2023

switch to being a systemd service containers/bootc#2

Open

cgwalters mentioned this pull request Dec 4, 2023

WIP: Use podman pull to fetch containers containers/bootc#215

Open

cgwalters mentioned this pull request Mar 25, 2024

install: Add --copy-container-credentials containers/bootc#428

Open

cgwalters mentioned this pull request May 3, 2024

bootc e2e workflow - build temp image shim with ssh keys around bootc containers/ai-lab-recipes#428

Merged

joelanford mentioned this pull request Sep 7, 2024

✨unpacker: switch from google/go-containerregistry to containers/image operator-framework/operator-controller#1194

Merged

5 tasks

joelanford mentioned this pull request Sep 23, 2024

[epic] Support installing bundles from a private registry operator-framework/operator-controller#1015

Closed

ipanova mentioned this pull request Oct 23, 2024

Follow up auth #2600

Draft

cgwalters closed this Oct 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Add a `/etc/containers/auth.json` #1746

WIP: Add a `/etc/containers/auth.json` #1746

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac left a comment •

edited

Loading

mtrmac Dec 8, 2022

mtrmac Dec 8, 2022

mtrmac Dec 8, 2022

cgwalters Jan 3, 2023

mtrmac Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 8, 2022 •

edited

Loading

mtrmac commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 9, 2022

cgwalters commented Dec 15, 2022

mtrmac commented Dec 16, 2022 •

edited

Loading

cgwalters commented Jan 3, 2023

cgwalters commented Jan 3, 2023

mtrmac Jan 4, 2023

cgwalters commented Feb 17, 2023

cgwalters commented Oct 17, 2023

cgwalters commented Mar 27, 2024

vrothberg commented Jun 26, 2024

rhatdan commented Jun 26, 2024

cgwalters commented Jun 26, 2024

mtrmac commented Jun 26, 2024

mtrmac commented Jun 26, 2024

cgwalters commented Jun 26, 2024

mtrmac commented Jun 27, 2024

cgwalters commented Oct 1, 2024

mtrmac commented Oct 1, 2024

cgwalters commented Oct 1, 2024

cgwalters commented Oct 23, 2024

WIP: Add a /etc/containers/auth.json #1746

WIP: Add a /etc/containers/auth.json #1746

Conversation

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac left a comment • edited Loading

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

cgwalters Jan 3, 2023

Choose a reason for hiding this comment

mtrmac Dec 8, 2022

Choose a reason for hiding this comment

cgwalters commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 8, 2022 • edited Loading

mtrmac commented Dec 8, 2022

cgwalters commented Dec 8, 2022

mtrmac commented Dec 9, 2022

cgwalters commented Dec 15, 2022

mtrmac commented Dec 16, 2022 • edited Loading

cgwalters commented Jan 3, 2023

cgwalters commented Jan 3, 2023

mtrmac Jan 4, 2023

Choose a reason for hiding this comment

cgwalters commented Feb 17, 2023

cgwalters commented Oct 17, 2023

cgwalters commented Mar 27, 2024

vrothberg commented Jun 26, 2024

rhatdan commented Jun 26, 2024

cgwalters commented Jun 26, 2024

mtrmac commented Jun 26, 2024

mtrmac commented Jun 26, 2024

cgwalters commented Jun 26, 2024

mtrmac commented Jun 27, 2024

cgwalters commented Oct 1, 2024

mtrmac commented Oct 1, 2024

cgwalters commented Oct 1, 2024

cgwalters commented Oct 23, 2024

WIP: Add a `/etc/containers/auth.json` #1746

WIP: Add a `/etc/containers/auth.json` #1746

mtrmac left a comment •

edited

Loading

mtrmac commented Dec 8, 2022 •

edited

Loading

mtrmac commented Dec 16, 2022 •

edited

Loading