Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Supports for Exports Read/Writing From S3 Buckets #1137

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

1nv8rzim
Copy link

@1nv8rzim 1nv8rzim commented Sep 19, 2024

Purpose

One of the things I have been working towards is a stateless deploy of yeti in order to deploy several replicas of the celery runner and api containers. The biggest blocker for this is that a shared volumes between several containers is an anti-pattern and not supported on the cluster I am deploying to.

This PR introduces the ability to replaces this shared volume with an S3 Complaint Bucket

Changes

  • When system.export_path is prefixed with s3:// it will attempt to use S3 as the storage mediums for exports
    • Example: Setting system.export_path to s3://bucket_name would use bucket_name for storage of export task results
  • Celery Runner can now upload results export tasks to an s3 compliant bucket
  • API is able to read results of export task from an s3 compliant bucket
  • Credentials and configuration for s3 buckets access should be injected by environment variables

@1nv8rzim
Copy link
Author

1nv8rzim commented Sep 19, 2024

This works; I am drafting a more generalized approach here so that this same thing can be extended past s3 in the future

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant