all: migrate aws-sdk-go to aws-sdk-go-v2 #3220
Comments
Having previously made a pull request removing azure-sdk-for-go as a dependency of gocloud.dev, this PR updates the CLI to remove the transitive dependency. Removing the dependency on azure-sdk-for-go reduces module cache weight by 492MiB. That's the apparent size of `~/go/pkg/mod/github.com/!azure` according to a disk usage utility. This makes the new largest targets for reducing size: ``` 258 MiB google.golang.org/[email protected] - required by 250 MiB github.com/aws/[email protected] - required by pulumi/pkg/operations, an experimental function, not used 83 MiB github.com/pierrec/[email protected]+incompatible - required by gocloud.dev/secrets/hashivault 25 MiB github.com/pulumi/pulumi-java ... ``` Returns rapidly diminish with most packages weighing under 1MiB after these. Follow up work ============== I've made an upstream issue with gocloud.dev to migrate off aws-sdk-go@v1: - google/go-cloud#3220 Moving storage backends to plugins would allow us to remove this and many other dependencies from the Pulumi CLI repository and remove conflicts between the Pulumi engine and providers. See: - pulumi/pulumi-vault#197
Having previously made a pull request removing azure-sdk-for-go as a dependency of gocloud.dev, this PR updates the CLI to remove the transitive dependency. Removing the dependency on azure-sdk-for-go reduces module cache weight by 492MiB. That's the apparent size of `~/go/pkg/mod/github.com/!azure` according to a disk usage utility. This makes the new largest targets for reducing size: ``` 258 MiB google.golang.org/[email protected] - required by 250 MiB github.com/aws/[email protected] - required by pulumi/pkg/operations, an experimental function, not used 83 MiB github.com/pierrec/[email protected]+incompatible - required by gocloud.dev/secrets/hashivault 25 MiB github.com/pulumi/pulumi-java ... ``` Returns rapidly diminish with most packages weighing under 1MiB after these. Follow up work ============== I've made an upstream issue with gocloud.dev to migrate off aws-sdk-go@v1: - google/go-cloud#3220 Moving storage backends to plugins would allow us to remove this and many other dependencies from the Pulumi CLI repository and remove conflicts between the Pulumi engine and providers. See: - pulumi/pulumi-vault#197
Having previously made a pull request removing azure-sdk-for-go as a dependency of gocloud.dev, this PR updates the CLI to remove the transitive dependency. Removing the dependency on azure-sdk-for-go reduces module cache weight by 492MiB. That's the apparent size of `~/go/pkg/mod/github.com/!azure` according to a disk usage utility. This makes the new largest targets for reducing size: ``` 258 MiB google.golang.org/[email protected] - required by 250 MiB github.com/aws/[email protected] - required by pulumi/pkg/operations, an experimental function, not used 83 MiB github.com/pierrec/[email protected]+incompatible - required by gocloud.dev/secrets/hashivault 25 MiB github.com/pulumi/pulumi-java ... ``` Returns rapidly diminish with most packages weighing under 1MiB after these. Follow up work ============== I've made an upstream issue with gocloud.dev to migrate off aws-sdk-go@v1: - google/go-cloud#3220 Moving storage backends to plugins would allow us to remove this and many other dependencies from the Pulumi CLI repository and remove conflicts between the Pulumi engine and providers. See: - pulumi/pulumi-vault#197
11963: chore: Update gocloud.dev, removing azure-sdk-for-go dep r=AaronFriel a=AaronFriel Having previously made a pull request removing azure-sdk-for-go as a dependency of gocloud.dev, this PR updates the CLI to remove the transitive dependency. Removing the dependency on azure-sdk-for-go reduces module cache weight by 492MiB. That's the apparent size of `~/go/pkg/mod/github.com/!azure` according to a disk usage utility. This makes the new largest targets for reducing size: ``` 258 MiB google.golang.org/[email protected] - required by 250 MiB github.com/aws/[email protected] - required by pulumi/pkg/operations, an experimental function, not used 83 MiB github.com/pierrec/[email protected]+incompatible - required by gocloud.dev/secrets/hashivault 25 MiB github.com/pulumi/pulumi-java ... ``` Returns rapidly diminish with most packages weighing under 1MiB after these. Follow up work ============== I've made an upstream issue with gocloud.dev to migrate off aws-sdk-go@v1: - google/go-cloud#3220 Moving storage backends to plugins would allow us to remove this and many other dependencies from the Pulumi CLI repository and remove conflicts between the Pulumi engine and providers. See: - pulumi/pulumi-vault#197 Co-authored-by: Aaron Friel <[email protected]>
|
Already done? E.g., https://pkg.go.dev/[email protected]/blob/s3blob#OpenBucketV2 https://pkg.go.dev/[email protected]/blob/s3blob#URLOpener |
|
Ah, my issue is intended to describe a migration that involves removing the aws-sdk-go dependency, as it's ~250MiB. As it stands, from a dependency size and security perspective right now we've got the worst of both. From a compatibility perspective, of course I understand needing to gracefully migrate clients over to awssdk=v2. |
|
Sorry, I'm not sure how to make that happen and maintain backwards compatibility.... and I'm not sure of the timeframe in which it would be OK to drop v1. |
|
That makes sense. I was looking at why CI takes so long to build a project and This is not an urgent issue, if you think it makes sense to close because it's premature feel free. I think this would be a good goal for some vNext in the future or a v1/v2 release of gocloud.dev. There are some really heavy packages in the dependency graph. |
|
This is a known issue and not likely to change, sorry. It's inherent in having a multi-cloud library that we're going to have dependencies from multiple cloud providers (and in some cases, multiple dependencies even from one cloud provider). If at some point AWS announces they are no longer supporting If you have ideas on how to reduce the dependency graph while still keeping support for multiple providers, let me know. I believe it is only a CI issue, at runtime Go won't load libraries you aren't using. |
|
You're right that it's primarily a CI and developer loop issue. Starting with a clean cache or when switching between projects using different versions of gocloud.dev causes large download times, poor developer experience. For what it's worth the PR you helped move across the finish line removed 500MiB of Azure dependencies. Perhaps it's worth revisiting: why do we have |
It can't though. The constructors and various I work for Google so I'm not super in touch with what AWS is doing; if there's some evidence that some high percentage of users have updated to the AWS V2 APIs or will be forced to do so soon that would be good to know. |
|
That makes sense, thanks! I'd still be interested in a future v1/v2 release that removed legacy cloud APIs, but understand that's far enough off that it doesn't make sense to track in this issue. |
|
@vangent Amazon has announced the end of support for v1. |
|
I created a new issue to track this. If any help is required on this, then please inform. |
Is your feature request related to a problem? Please describe.
Migrating to the new SDK places the project on surer footing, using the currently supported SDK and reduces module weight. See: https://aws.github.io/aws-sdk-go-v2/docs/migrating/
Describe the solution you'd like
Updates to packages such as blob/s3blob to use the new SDK.
Describe alternatives you've considered
n/a - alternatives are absurd, i.e.: reimplement AWS SDK from scratch.
Additional context
The AWS SDK for Go is still in GA, that is, there is no immediate risk of the aws-sdk-go v1 beocming unsupported.
That said, the newer SDK reduces dependency weight and is likely to be more actively maintained.
The text was updated successfully, but these errors were encountered: