Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

blob/s3blob: s3ForcePathStyle stopped working in v0.39.0 #3472

Closed
caarlos0 opened this issue Aug 18, 2024 · 9 comments · Fixed by #3505
Closed

blob/s3blob: s3ForcePathStyle stopped working in v0.39.0 #3472

caarlos0 opened this issue Aug 18, 2024 · 9 comments · Fixed by #3505

Comments

@caarlos0
Copy link
Contributor

caarlos0 commented Aug 18, 2024 •
edited
Loading

Describe the bug

This was flagged by goreleaser's tests when dependabot opened the pr updating from v0.38 to v0.39.

I'm not sure if this is expected and just not documented, or if it is in fact a bug.

To Reproduce

open a bucket (can be a local minio for example) using s3ForcePathStyle.

Expected behavior

To work.

Version

v0.39.0

Additional context

Error:

open bucket s3://basic?endpoint=http%3A%2F%2Flocalhost%3A32812&region=us-east&s3ForcePathStyle=true: unknown query parameter "s3ForcePathStyle"

Also, forcing awssdk=v1 makes it work again.
This was referenced Aug 18, 2024
Closed
Merged
@vangent
Copy link
Contributor

vangent commented Aug 18, 2024

Please see the release notes (https://github.com/google/go-cloud/releases/tag/v0.39.0). This release changed the default for AWS URLs from their v1 SDK to their v2 SDK. Some parameters are different, s3ForcePathStyle is one that doesn't seem to exist for v2.

You should either update the URL to work with v2 (possibly just dropping that parameter will fix it?), or add awssdk=v1 (which will work for now, but support will be dropped at some point, the release notes proposed 6 months).

caarlos0 added a commit to goreleaser/goreleaser that referenced this issue Aug 18, 2024
@caarlos0
chore(deps): update gocloud to v0.39.0 (#5091)
c95bcdc 
forcing `awssdk=v1` for now.

will probably need to remove that soon, holding it now only because of
google/go-cloud#3472

Signed-off-by: Carlos Alexandro Becker <[email protected]>
@caarlos0
Copy link
Contributor Author

yes, I saw the release notes, and also that forcing v1 works, my point was that the specific breaking change is not documented anywhere.

also, just dropping that part of the URL doesn't work (at least not with minio).

@vangent
Copy link
Contributor

vangent commented Aug 19, 2024

OK. Yes, I didn't try to document every specific URL parameter that wouldn't work anymore, just "Most URLs should continue to work, but in some cases you may need to add awssdk=v1 to force V1 explicitly."

@vangent vangent closed this as not planned Won't fix, can't repro, duplicate, stale Aug 19, 2024
@vangent
Copy link
Contributor

vangent commented Aug 19, 2024

just dropping that part of the URL doesn't work (at least not with minio

Please try to figure out what's needed to get V2 to work with Minio (I'm not sure). It's possible we need to add support for more URL parameters for V2.

caarlos0 added a commit to caarlos0/go-cloud that referenced this issue Aug 19, 2024
@caarlos0
fix: custom endpoints with s3 and aws sdk v2
d7df96f 
refs google#3472

Signed-off-by: Carlos Alexandro Becker <[email protected]>
@caarlos0 caarlos0 mentioned this issue Aug 19, 2024
Merged
@vangent vangent mentioned this issue Aug 19, 2024
Closed
@dmotylev
Copy link

dmotylev commented Oct 9, 2024

It seems s3ForcePathStyle has been moved to https://github.com/aws/aws-sdk-go-v2/blob/0cbb5aa17f9078cb45dc0e82d3e1d0abee3744a9/service/s3/options.go#L147-L150

@vangent
Copy link
Contributor

vangent commented Oct 9, 2024

Added, see #3491.

@stanhu
Copy link
Contributor

stanhu commented Nov 7, 2024 •
edited
Loading

@vangent Would you consider adding an alias for s3ForcePathStyle with use_path_style for backwards compatibility? We've had a few customers hit backup failures due to this URL change after we upgraded to v0.39.0.

@vangent
Copy link
Contributor

vangent commented Nov 7, 2024

Sure, that sounds reasonable; will send a PR.

@vangent vangent reopened this Nov 7, 2024
@vangent vangent mentioned this issue Nov 7, 2024
Merged
@stanhu
Copy link
Contributor

stanhu commented Nov 7, 2024

@vangent Thanks so much!

gitlab-runner-bot pushed a commit to gitlabhq/gitlab-runner that referenced this issue Nov 14, 2024
@stanhu
Fix path-style requests with Upload ARN functionality
15e4e15 
Previously only the `hostname_immutable` flag was set on the GoCloud
URL for AWS when the upload ARN were used. This is not sufficent.
As explained in google/go-cloud#3472,
we also need the `use_path_style` query parameter.

This should not affect most buckets because path-style access only
works on buckets before September 2020
(https://aws.amazon.com/blogs/aws/amazon-s3-path-deprecation-plan-the-rest-of-the-story/).
However, in the oft chance we have customers still using path-style
access with S3 with the newly-introduced Upload ARN functionality,
then this change will allow that to work.

Changelog: fixed
@loicalleyne loicalleyne mentioned this issue Dec 12, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@dmotylev @caarlos0 @stanhu @vangent