Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add CloudFront invalidation to SFA S3 upload workflow #5

Draft
wants to merge 2 commits into
base: main
Choose a base branch
from
Draft

Add CloudFront invalidation to SFA S3 upload workflow #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
17e813c
Add CloudFront invalidation to SFA forecast upload workflow
davereinhart Mar 21, 2024
189835b
Update documentation related to final output, S3 and CloudFront workflow
davereinhart Mar 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion forecasts-ncov-states.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,4 +44,4 @@ An ECS task definition specifies the environment and nexstrain CLI command to ru

### Final S3 outputs

To update the files available from the public website, an addition workflow rule can be run by appending `config/sfa-optional.yaml` to the existing `--configfile` argument. The config file specifies the destination bucket, triggers the additional workflow that compresses and copies the files from the source bucket to the destination, and sets the appropriate encoding and content types for zipped files to be served via CloudFront. (i.e. `content-encoding:gzip and content-type:application/json`)
To update the files on the public website, an addition workflow rule is run by setting `send_sfa_forecast_to_s3` to `True` in config.yaml. The destination bucket and associated CloudFront distribution must be set via environment variables `sfa_s3_bucket` and `sfa_s3_bucket_cloudfront_id`. This workflow compresses and copies the files from the source bucket to the destination, sets the appropriate encoding and content types for pre-zipped files to be served via CloudFront (i.e. `content-encoding:gzip and content-type:application/json`), and invalidates the `/data/*` path of the CloudFront distribution.
10 changes: 10 additions & 0 deletions workflow/snakemake_rules/upload_sfa_forecast.smk
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,13 @@ rule upload_zip_files_to_s3:
aws s3api put-object --bucket {params.s3_bucket} --key {params.us_s3_key} --content-encoding gzip --content-type application/json --body {input.us_results} && \
aws s3api put-object --bucket {params.s3_bucket} --key {params.wa_s3_key} --content-encoding gzip --content-type application/json --body {input.wa_results}
"""

rule cloudfront_invalidation:
input: "results/{data_provenance}/{variant_classification}/usa/{model}/{date}_s3_upload.done"
output: touch("results/{data_provenance}/{variant_classification}/usa/{model}/{date}_cloudfront_invalidation.done")
params:
cloudfront_id = os.environ["sfa_s3_bucket_cloudfront_id"]
shell:
"""
aws cloudfront create-invalidation --distribution-id {params.cloudfront_id} --paths '/data/*'
"""