Oauth2 support without Python #626

lukaszwawrzyk · 2024-06-12T12:03:00Z

Due to inactivity, I picked up this PR: #581

I implememented the mentioned workarounds, i.e. in case of windows, I try to run script using powershell, otherwise, I fallback to curl and then wget, both wrapped in a bash script.

I tested these with an azure repo which requires oauth2. I also tested it on windows. I found that the config.json is not found due to "stat" binary not existing on windows, so I fixed this too.

oci/private/util.bzl

oci/private/authn.bzl

thesayyn · 2024-06-12T17:58:28Z

Okay, this looks a bit better, this could use some tests similar to https://github.com/bazel-contrib/rules_oci/blob/2.x/e2e/assertion/oci_pull_auth_tests.bats

We will also need better error messages.

Sowmya-Iyer · 2024-06-12T19:25:28Z

While trying to oci.pull an azure container registry, I still get this error:

Error in download: java.io.IOException: Error downloading [<>] to /home/siyer/.cache/bazel/_bazel_siyer/c0c06e0bd45258079fb49c5a3cba7fd4/external/rules_oci~~oci~ard_x86_64_base/www-authenticate.json: GET returned 401 Unauthorized

The PR #600 had actually fixed that issue for me though I didn't like the way it was done either

thesayyn · 2024-06-12T19:28:54Z

we should somehow incorporate that change here. identity tokens are not something we support at the moment, so we might as well it right at once and improve on it.

lukaszwawrzyk · 2024-06-13T09:27:44Z

While trying to oci.pull an azure container registry, I still get this error:
Error in download: java.io.IOException: Error downloading [<>] to /home/siyer/.cache/bazel/_bazel_siyer/c0c06e0bd45258079fb49c5a3cba7fd4/external/rules_oci~~oci~ard_x86_64_base/www-authenticate.json: GET returned 401 Unauthorized 
The PR #600 had actually fixed that issue for me though I didn't like the way it was done either

If I az acr login --name <my repo> from terminal and then run bazel with changes in this PR it works. While it doesn't work without these changes at all.

This reverts commit 3e7fb4a.

lukaszwawrzyk · 2024-06-13T12:39:54Z

Okay, this looks a bit better, this could use some tests similar to https://github.com/bazel-contrib/rules_oci/blob/2.x/e2e/assertion/oci_pull_auth_tests.bats

We will also need better error messages.

I think I fixed everything requested except for tests.

Current tests do not pass for me so it is hard to develop new, any suggestions?
I run them like this:

% pwd                                                                                                                                                             
/Users/lukasz/dev/rules/rules_oci/e2e/pull
% bazel test //:test_nobzlmod

And this is the test log that I get:
test.log

Sowmya-Iyer · 2024-06-14T13:23:41Z

While trying to oci.pull an azure container registry, I still get this error:
Error in download: java.io.IOException: Error downloading [<>] to /home/siyer/.cache/bazel/_bazel_siyer/c0c06e0bd45258079fb49c5a3cba7fd4/external/rules_oci~~oci~ard_x86_64_base/www-authenticate.json: GET returned 401 Unauthorized 
The PR #600 had actually fixed that issue for me though I didn't like the way it was done either
If I az acr login --name <my repo> from terminal and then run bazel with changes in this PR it works. While it doesn't work without these changes at all.

I added common --repo_env=OCI_ENABLE_OAUTH2_SUPPORT=1 to my bazelrc
uodated Module.bzl with this commit as a git_override (4f78d52)
az acr login --name <container_registry>

I ran the bazel with the above three changes

ERROR:

File "/home/siyer/.cache/bazel/_bazel_siyer/c0c06e0bd45258079fb49c5a3cba7fd4/external/rules_oci~/oci/private/authn.bzl", line 304, column 30, in _get_token
		rctx.download(
Error in download: java.io.IOException: Error downloading [<redacted>] to /home/siyer/.cache/bazel/_bazel_siyer/c0c06e0bd45258079fb49c5a3cba7fd4/external/rules_oci~~oci~<image_name>/www-authenticate.json: GET returned 401 Unauthorized

For my case, it doesn't seem to reach line 287, because I did not get the fail(IDENTITY_TOKEN_WARNING) without the bazelrc changes

lukaszwawrzyk · 2024-06-14T14:36:56Z

I am not sure if the container registry that you are using is compatible with this fix or what is wrong in your case.

Firstly, I will post my reproduction if it can help on any way:

lukasz@vl-d-1274 test_rules_oci % bazel build @azur//:azur
Starting local Bazel server and connecting to it...
INFO: Repository azur instantiated at:
  /Users/lukasz/dev/rules/test_rules_oci/WORKSPACE:37:9: in <toplevel>
  /private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/pull.bzl:212:14: in oci_pull
Repository rule oci_alias defined at:
  /private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/pull.bzl:370:28: in <toplevel>
ERROR: An error occurred during the fetch of repository 'azur':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/pull.bzl", line 304, column 55, in _oci_alias_impl
		manifest, _, digest = downloader.download_manifest(rctx.attr.identifier, "mf.json")
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/pull.bzl", line 163, column 74, in lambda
		download_manifest = lambda identifier, output: _download_manifest(rctx, authn, identifier, output),
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/pull.bzl", line 123, column 23, in _download_manifest
		result = _download(rctx, authn, identifier, output, "manifests", allow_fail = True)
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/pull.bzl", line 89, column 27, in _download
		auth = authn.get_token(rctx.attr.registry, rctx.attr.repository)
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/authn.bzl", line 356, column 49, in lambda
		get_token = lambda reg, repo: _get_token(rctx, state, reg, repo),
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/authn.bzl", line 290, column 35, in _get_token
		response = _oauth2(
	File "/private/var/tmp/_bazel_lukasz/5b7ad260b35538c8633543ce918f5e2d/external/rules_oci/oci/private/authn.bzl", line 205, column 13, in _oauth2
		fail("oauth2 failed:\nSTDOUT:\n{}\nSTDERR:\n{}".format(result.stdout, result.stderr))
Error in fail: oauth2 failed:
STDOUT:

STDERR:
curl: (22) The requested URL returned error: 403
ERROR: oauth2 failed:
STDOUT:

STDERR:
curl: (22) The requested URL returned error: 403
INFO: Elapsed time: 7.087s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
lukasz@vl-d-1274 test_rules_oci %
lukasz@vl-d-1274 test_rules_oci % az acr login --name dockerregistrytest99
Login Succeeded
lukasz@vl-d-1274 test_rules_oci % bazel build @azur//:azur
WARNING: Fetching from hello-world@v1 without an integrity hash. The result will not be cached.
WARNING: The registry sent a manifest with schemaVersion=1.
This commonly occurs when fetching from a registry that needs the Docker-Distribution-API-Version header to be set.

WARNING: The use of Curl fallback is deprecated and is set to be removed in version 2.0.
For more details, refer to: https://github.com/bazel-contrib/rules_oci/issues/456

WARNING: Fetching from hello-world@v1 without an integrity hash. The result will not be cached.
WARNING: Fetching from hello-world@v1 without an integrity hash. The result will not be cached.
WARNING: The registry sent a manifest with schemaVersion=1.
This commonly occurs when fetching from a registry that needs the Docker-Distribution-API-Version header to be set.

WARNING: The use of Curl fallback is deprecated and is set to be removed in version 2.0.
For more details, refer to: https://github.com/bazel-contrib/rules_oci/issues/456

WARNING: Fetching from hello-world@v1 without an integrity hash. The result will not be cached.
INFO: Analyzed target @@azur//:azur (87 packages loaded, 458 targets configured).
INFO: Found 1 target...
Target @@azur_single//:azur_single up-to-date:
  bazel-bin/external/azur_single/layout
INFO: Elapsed time: 6.932s, Critical Path: 0.02s
INFO: 1 process: 1 internal.
INFO: Build completed successfully, 1 total action
lukasz@vl-d-1274 test_rules_oci % cat WORKSPACE
workspace(name = "my_workspace")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

local_repository(
    name = "rules_oci",
    path = "../rules_oci",
)

load("@rules_oci//oci:dependencies.bzl", "rules_oci_dependencies")

rules_oci_dependencies()

load("@rules_oci//oci:repositories.bzl", "LATEST_CRANE_VERSION", "oci_register_toolchains")

oci_register_toolchains(
    name = "oci",
    crane_version = LATEST_CRANE_VERSION,
    # Uncommenting the zot toolchain will cause it to be used instead of crane for some tasks.
    # Note that it does not support docker-format images.
    # zot_version = LATEST_ZOT_VERSION,
)

# You can pull your base images using oci_pull like this:
load("@rules_oci//oci:pull.bzl", "oci_pull")

oci_pull(
    name = "distroless_base",
    digest = "sha256:ccaef5ee2f1850270d453fdf700a5392534f8d1a8ca2acda391fbb6a06b81c86",
    image = "gcr.io/distroless/base",
    platforms = [
        "linux/amd64",
        "linux/arm64",
    ],
)

oci_pull(
  name = "azur",
  image = "dockerregistrytest99.azurecr.io/hello-world",
  tag = "v1",
  reproducible = False,
)%
lukasz@vl-d-1274 test_rules_oci % cat .bazelrc
common --repo_env=OCI_ENABLE_OAUTH2_SUPPORT=1%
lukasz@vl-d-1274 test_rules_oci % cd ../rules_oci
lukasz@vl-d-1274 rules_oci % git rev-parse HEAD
4f78d52446b074a377d7e12a1baf7ac8072a7343

First I get 403, my azure subscription was actually disabled, then I enabled it and logged in successfully, then I can download the image. In the error stack trace you can also see that it enters the new code. You can see details of my setup with further commands.

For you, new code is not used at all as you mentioned. The condition if pattern["login"] == "<token>" must be false for you.

For me, it goes to line 234, i.e. calls _fetch_auth_via_creds_helper based on my empty dict:

$ cat ~/.docker/config.json
{
	"auths": {
		"dockerregistrytest99.azurecr.io": {},
		"ghcr.io": {},
	},
	"credsStore": "desktop",
	"currentContext": "desktop-linux"
}

and then calls

$ docker-credential-desktop get <<< dockerregistrytest99.azurecr.io
{"ServerURL":"dockerregistrytest99.azurecr.io","Username":"\u003ctoken\u003e","Secret":"<redacted>"}

and based on Username key, login becomes <token> and the condition is true. We verified it on 2 different azure accounts (prod and the dummy test one that I created and use here) by 3 people on different macbooks.

I am curious why it works differently for you, you might want to trace how credentials are obtained for you and compare with what I have written here.

Sowmya-Iyer · 2024-06-14T15:44:33Z

So, I think the problem is in parsing the ~/.docker/config.json:

siyer@siyer-ThinkPad-T15g-Gen-2i:~/workspace/container/BuildFarm$ cat ~/.docker/config.json
{
	"auths": {
		"<registry>.azurecr.io": {
			"auth": "MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAwOg==",
			"identitytoken": "<redacted>"
		}
	}
}

Line 240 looks for what follows ":" as password but as per this config it is ["identity_token"] . Since here auth is weirdly ending with ":" 00000000-0000-0000-0000-000000000000: , the "if not sep" condition fails and there is no error message and password is empty as well

oci/private/authn.bzl

lukaszwawrzyk · 2024-06-17T13:30:16Z

So, I think the problem is in parsing the ~/.docker/config.json:
siyer@siyer-ThinkPad-T15g-Gen-2i:~/workspace/container/BuildFarm$ cat ~/.docker/config.json
{
	"auths": {
		"<registry>.azurecr.io": {
			"auth": "MDAwMDAwMDAtMDAwMC0wMDAwLTAwMDAtMDAwMDAwMDAwMDAwOg==",
			"identitytoken": "<redacted>"
		}
	}
}
Line 240 looks for what follows ":" as password but as per this config it is ["identity_token"] . Since here auth is weirdly ending with ":" 00000000-0000-0000-0000-000000000000: , the "if not sep" condition fails and there is no error message and password is empty as well

My best bet is that you are using linux, I managed to reproduce the issue on ubuntu. So the fix in the current PR works for Mac OS, but not for linux.

lukaszwawrzyk · 2024-06-17T13:52:29Z

I integrated fix from @ddeville to this PR, I believe we now have a complete support of azure across operating systems. And for oauth2 hopefully in all cases.

thesayyn

This looks fantastic now. Only we knew how to add some tests for this.

thesayyn · 2024-06-20T14:03:05Z

I am going to hold this for a little bit so we can figure out some testing infra for this. i'll circle back to this next week.

thesayyn · 2024-07-08T13:54:30Z

Hmm, there are some failures. Can you handle those?

lukaszwawrzyk · 2024-07-09T09:25:12Z

@thesayyn it was a simple check missing, now it's all good

wingsofovnia · 2024-07-17T10:27:20Z

Great work! Looking forward to having it in the next releases.
Any chance @thesayyn we can get this merged for 1.8.1?

wingsofovnia · 2024-07-18T13:19:04Z

Thank you. @thesayyn will this also get pulled to 2.x?

Co-authored-by: xbog300 <[email protected]>

mpatou-openai · 2024-08-21T18:09:13Z

I'm still seeing issues with azure actually. It might work if the login is <token> but I have 00000000-0000-0000-0000-000000000000 as login.

xbog300 and others added 3 commits May 20, 2024 16:55

add oauth2 support

b560bc1

use curl, wget and powershell instead of python

07ac43f

Fix finding files on windows

3e7fb4a

lukaszwawrzyk mentioned this pull request Jun 12, 2024

add OAuth2 support #581

Closed

thesayyn requested changes Jun 12, 2024

View reviewed changes

thesayyn mentioned this pull request Jun 12, 2024

Support Azure identity token in docker config #600

Closed

lukaszwawrzyk added 4 commits June 13, 2024 11:29

Revert "Fix finding files on windows"

9b27a26

This reverts commit 3e7fb4a.

Extract scripts into global constants

524083a

Hide identity token support behind a flag

6c668f0

Remove redundant error message lines

4f78d52

Add compatibility for bazel < 7

a28a952

thesayyn reviewed Jun 17, 2024

View reviewed changes

oci/private/authn.bzl Outdated Show resolved Hide resolved

Use rctx.os.environ for compatibility

5fb2b16

Support for azure on linux (by @ddeville)

8ac0e84

thesayyn approved these changes Jun 19, 2024

View reviewed changes

Add safeguard for accessing 'login' key

e2e54ea

thesayyn merged commit 9948ee1 into bazel-contrib:main Jul 17, 2024
20 checks passed

This was referenced Jul 18, 2024

FR: oci_pull support identity tokens #129

Closed

oci_pull fails with authorization error #475

Open

thesayyn pushed a commit that referenced this pull request Jul 25, 2024

feat: support Oauth2 for oci_pull (#626)

44edb69

Co-authored-by: xbog300 <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Oauth2 support without Python #626

Oauth2 support without Python #626

lukaszwawrzyk commented Jun 12, 2024

thesayyn commented Jun 12, 2024

Sowmya-Iyer commented Jun 12, 2024

thesayyn commented Jun 12, 2024

lukaszwawrzyk commented Jun 13, 2024

lukaszwawrzyk commented Jun 13, 2024

Sowmya-Iyer commented Jun 14, 2024 •

edited

Loading

lukaszwawrzyk commented Jun 14, 2024 •

edited

Loading

Sowmya-Iyer commented Jun 14, 2024 •

edited

Loading

lukaszwawrzyk commented Jun 17, 2024

lukaszwawrzyk commented Jun 17, 2024

thesayyn left a comment

thesayyn commented Jun 20, 2024

thesayyn commented Jul 8, 2024

lukaszwawrzyk commented Jul 9, 2024

wingsofovnia commented Jul 17, 2024

wingsofovnia commented Jul 18, 2024

mpatou-openai commented Aug 21, 2024

Oauth2 support without Python #626

Oauth2 support without Python #626

Conversation

lukaszwawrzyk commented Jun 12, 2024

thesayyn commented Jun 12, 2024

Sowmya-Iyer commented Jun 12, 2024

thesayyn commented Jun 12, 2024

lukaszwawrzyk commented Jun 13, 2024

lukaszwawrzyk commented Jun 13, 2024

Sowmya-Iyer commented Jun 14, 2024 • edited Loading

lukaszwawrzyk commented Jun 14, 2024 • edited Loading

Sowmya-Iyer commented Jun 14, 2024 • edited Loading

lukaszwawrzyk commented Jun 17, 2024

lukaszwawrzyk commented Jun 17, 2024

thesayyn left a comment

Choose a reason for hiding this comment

thesayyn commented Jun 20, 2024

thesayyn commented Jul 8, 2024

lukaszwawrzyk commented Jul 9, 2024

wingsofovnia commented Jul 17, 2024

wingsofovnia commented Jul 18, 2024

mpatou-openai commented Aug 21, 2024

Sowmya-Iyer commented Jun 14, 2024 •

edited

Loading

lukaszwawrzyk commented Jun 14, 2024 •

edited

Loading

Sowmya-Iyer commented Jun 14, 2024 •

edited

Loading