s3blob: Need a way to set custom credentials provider for S3 when using v2

[hairyhenderson]

Describe the bug

The aws-sdk-go (v1) module is being removed soon, and so I'm trying to migrate to aws-sdk-go-v2 (see also #3489).

I have a need to sometimes use the anonymous CredentialsProvider, and with V1 I could achieve this by setting the ConfigProvider field with an AWS session properly configured with the anonymous provider.

I'm using the blob.BucketURLOpener interface to support non-S3 buckets as well via URLs, and so using s3blob.OpenBucketV2 with a custom client isn't an option.

To Reproduce

Attempt to use &s3blob.URLOpener{UseV2: true} to open S3 buckets with anonymous credential...

Expected behavior

There are a few ways to achieve this - one way would be to allow adding additional config.LoadOptions items to be provided to s3blob.V2ConfigFromURLParams via a new field in the s3blob.URLOpener struct.

Alternately, a new URL query parameter could be supported - something like anonymous=true. FWIW, that would be useful in the gcsblob package as well.

Version

v0.40.0 (latest)

Additional context

n/a

[vangent]

I will take a look. It would save me some time if you could provide some code snippets for how you use anonymous credential with AWSv1 + Go CDK today, and how you would access S3 with anonymous credentials without Go CDK. Thanks!

[hairyhenderson]

@vangent if you have a look at the diff in hairyhenderson/go-fsimpl#892 you'll see the old v1 approach and the new one - I copied some of the code from here to override it temporarily and support an anonymous query param.

Let me know if that's useful, or if you want more explanation...

[vangent]

Can you take a look at #3513 and see if it meets your needs? I've added an "anonymous" query parameter for "gcsblob" (previously you could get the same behavior with access_id=- FWIW), and also an "anonymous" query parameter that should work for all of the AWSv2 providers.

[hairyhenderson]

@vangent perfect, thanks! and - I didn't know about access_id=-, though there's definitely value in having a shared parameter where possible.