Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⚡ Add support for high availability S3 backend #29191

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

danquack
Copy link

@danquack danquack commented Dec 13, 2024

Description

This PR prototypes state locking for Vault's s3 backend using Amazon S3’s recently introduced conditional writes feature to implement a locking mechanism.

Currently S3 does not implement the physical.HABackend to support high availability. With S3 Conditional writes, the if-none-match header can be applied during the Lock PutObject, which will enforce a remote lock state. In order to introduce this, an upgrade from sdk v1 to sdkv2 was needed. As a result, some work toward #18375 could be achieved due to how the configuration is loaded. This only applies to storage, as additional work would be needed to fully close that issue.

Tests

# go test -v ./physical/s3/...                                                                                                                                  
=== RUN   TestHABackend
2024-12-13T16:15:22.108-0500 [DEBUG] using other creds
2024-12-13T16:15:22.225-0500 [DEBUG] max_parallel set: max_parallel=0
2024-12-13T16:15:22.225-0500 [DEBUG] using other creds
2024-12-13T16:15:22.343-0500 [DEBUG] max_parallel set: max_parallel=0
--- PASS: TestHABackend (12.90s)
=== RUN   TestDefaultS3Backend
2024-12-13T16:15:34.729-0500 [DEBUG] added environment variable credential provider
2024-12-13T16:15:34.729-0500 [DEBUG] added shared credential provider
2024-12-13T16:15:35.061-0500 [DEBUG] using other creds
2024-12-13T16:15:35.247-0500 [DEBUG] max_parallel set: max_parallel=0
--- PASS: TestDefaultS3Backend (2.65s)
=== RUN   TestS3BackendSseKms
2024-12-13T16:15:37.376-0500 [DEBUG] added environment variable credential provider
2024-12-13T16:15:37.376-0500 [DEBUG] added shared credential provider
2024-12-13T16:15:37.705-0500 [DEBUG] using other creds
2024-12-13T16:15:37.863-0500 [DEBUG] max_parallel set: max_parallel=0
--- PASS: TestS3BackendSseKms (2.30s)
PASS
ok  	github.com/hashicorp/vault/physical/s3	18.084s

TODO only if you're a HashiCorp employee

  • Backport Labels: If this fix needs to be backported, use the appropriate backport/ label that matches the desired release branch. Note that in the CE repo, the latest release branch will look like backport/x.x.x, but older release branches will be backport/ent/x.x.x+ent.
    • LTS: If this fixes a critical security vulnerability or severity 1 bug, it will also need to be backported to the current LTS versions of Vault. To ensure this, use all available enterprise labels.
  • ENT Breakage: If this PR either 1) removes a public function OR 2) changes the signature
    of a public function, even if that change is in a CE file, double check that
    applying the patch for this PR to the ENT repo and running tests doesn't
    break any tests. Sometimes ENT only tests rely on public functions in CE
    files.
  • Jira: If this change has an associated Jira, it's referenced either
    in the PR description, commit message, or branch name.
  • RFC: If this change has an associated RFC, please link it in the description.
  • ENT PR: If this change has an associated ENT PR, please link it in the
    description. Also, make sure the changelog is in this PR, not in your ENT PR.

@danquack danquack requested review from a team as code owners December 13, 2024 21:10
@danquack danquack requested a review from charlesn-hc December 13, 2024 21:10
Copy link

hashicorp-cla-app bot commented Dec 13, 2024

CLA assistant check
All committers have signed the CLA.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants