Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(storage): AWS backend using thanos.io/objstore (backport k226) #14626

Merged
merged 1 commit into from
Oct 28, 2024

Conversation

loki-gh-app[bot]
Copy link
Contributor

@loki-gh-app loki-gh-app bot commented Oct 28, 2024

Backport b872246 from #11221


What this PR does / why we need it:

This PR adds support to use the thanos.io/objstore backend for the AWS provider for all components excluding the Ruler since it's similar to the work done in #11132.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

  • We tried to the best of our efforts to preserve feature parity so the new backend still uses the ObjectClient interface
  • Fix how some CLI variables we being passed (some HTTP CLI variables were under .http while others weren't)
  • Thanos.io/objstore doesn't support all the CLI flags Loki supports nowadays (see snippet below)
  • This PR was inspired by the way Mimir uses thanos.io/objstore with the goal of making this part of the codebase as similar as possible

CLI Table

s3.bucket (new, before it was buckets)
s3.endpoint
s3.region
s3.access-key-id
s3.secret-access-key
s3.session-token
s3.insecure
s3.storage-class 
s3.sse
s3.http.idle-conn-timeout
s3.http.response-header-timeout
s3.http.insecure-skip-verify
s3.http.ca-file
s3.signature-version (possible but I've deprecated it)

NOT PRESENT AT THE MOMENT
s3.url (replaced by endpoint)
s3.force-path-style (deprecated since we use endpoint)
s3.buckets
s3.http.timeout (only possible if we were to always pass full transport)
s3.min-backoff (Would need objstore patch)
s3.max-backoff (Would need objstore patch)
s3.max-retries (Would need objstore patch)

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
    • If the change is worth mentioning in the release notes, add add-to-release-notes label
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Co-authored-by: Ashwanth Goli <[email protected]>
(cherry picked from commit b872246)
@loki-gh-app loki-gh-app bot requested a review from a team as a code owner October 28, 2024 06:30
@loki-gh-app loki-gh-app bot added backport size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories labels Oct 28, 2024
Copy link
Contributor

@ashwanthgoli ashwanthgoli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ashwanthgoli ashwanthgoli merged commit 90c5d51 into k226 Oct 28, 2024
57 of 64 checks passed
@ashwanthgoli ashwanthgoli deleted the backport-11221-to-k226 branch October 28, 2024 06:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport size/XL type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants