You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Right now we support S3 and GCS as cloud storage providers and file upload/download is done directly in export DAG. Instead 2 classes S3CloudStorage and GCSCloudStorage can be created with common interface to abstract upload/download logic.
As an example:
class S3CloudStorage:
def download_file(...)
def upload_file(...)
upload_to_gcs and download_from_gcs can be moved from export DAG to GCSCloudStorage.
The text was updated successfully, but these errors were encountered:
medvedev1088
changed the title
Create CloudStorage class to abstract file upload/download
Create CloudStorage classes to abstract file upload/download
Mar 11, 2019
Right now we support S3 and GCS as cloud storage providers and file upload/download is done directly in export DAG. Instead 2 classes S3CloudStorage and GCSCloudStorage can be created with common interface to abstract upload/download logic.
As an example:
upload_to_gcs
anddownload_from_gcs
can be moved from export DAG to GCSCloudStorage.The text was updated successfully, but these errors were encountered: