Support#457: Add Flightpath

[BrianXu20]

Ticket: https://torchbox.monday.com/boards/1343847472/pulses/1343866317

This adds Flightpath to allow copy production data to development and also adds some new processors.

Birdbath:

• Update base.py
• Update docker-compose.yml
• Update fabfile.py

Flightpath:

• Create management command

Heroku:

Production

• BIRDBATH_REQUIRED=false
• dyno metadata enabled

Development

• dyno metadata enabled

Audit Site: (custom processors)

• ScholarshipEnquiryFormSubmission
• ScholarshipEnquiryFormSubmission
• StudentAccountAnonymiser (includes StudentPage and related records + Collections and User Groups)

Docs:

• Docs Updated

Tasks:

• Code Review
• Deploy to staging
• Staging reset
• Staging testing
• Deploy to production

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 4.06504% with 118 lines in your changes are missing coverage. Please review.

Project coverage is 52.12%. Comparing base (086956b) to head (0570c34).

Files	Patch %	Lines
rca/users/birdbath.py	0.00%	56 Missing ⚠️
...ca/utils/management/commands/_base_copy_command.py	0.00%	45 Missing ⚠️
fabfile.py	0.00%	4 Missing ⚠️
rca/enquire_to_study/birdbath.py	0.00%	4 Missing ⚠️
rca/scholarships/birdbath.py	0.00%	4 Missing ⚠️
rca/utils/management/commands/copy_db_to_dev.py	0.00%	4 Missing ⚠️
rca/settings/production.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #871      +/-   ##
==========================================
- Coverage   52.86%   52.12%   -0.74%     
==========================================
  Files         163      168       +5     
  Lines        7856     7977     +121     
  Branches      175      175              
==========================================
+ Hits         4153     4158       +5     
- Misses       3577     3693     +116     
  Partials      126      126              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nickmoreton]

Hey @BrianXu20 With my limited knowledge here the script looks OK.

I think Kevin H may be the person to contact about testing etc. he has helped me with circle CI on this project recently.

[nickmoreton]

Hey @BrianXu20 With my limited knowledge here the script looks OK.

I think Kevin H may be the person to contact about testing etc. he has helped me with circle CI on this project recently.

[BrianXu20]

I have tested this is working on my local machine, and it copied the prod data directly.

====>> Send POST request to FlightPath
  #!/bin/bash -eo pipefail
export JOB_STATUS=$(curl -f -X POST -m 900 -H "Authorization: Token $FLIGHTPATH_AUTH_KEY" -d "source_key=$DEPLOYMENT_KEY&destination_key=$DEPLOYMENT_KEY&copy_media=1&from_backup=1" "https://flightpath.torchbox.com/copy/$HEROKU_APP_NAME_PRODUCTION/$HEROKU_APP_NAME_STAGING/")
echo $JOB_STATUS
export JOB_ID=$(echo $JOB_STATUS | jq -r '.job_id')
echo $JOB_ID

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   561  100   406  100   155    167     64  0:00:02  0:00:02 --:--:--   167
{"time_started":"2022-05-25T05:35:03.936973Z","job_id":"ec2df4ec-ffb6-48fa-aa69-8da838690da7","status":"started","metadata":{"source_app":"rca-production","destination_app":"rca-development","copy_media":true,"create_snapshot":false,"from_backup":true,"source_key":"xxx","destination_key":"xxx"},"log_output":""}
ec2df4ec-ffb6-48fa-aa69-8da838690da7
Success!

[BrianXu20]

I have tested this is working on my local machine,

====>> Send POST request to FlightPath
  #!/bin/bash -eo pipefail
export JOB_STATUS=$(curl -f -X POST -m 900 -H "Authorization: Token $FLIGHTPATH_AUTH_KEY" -d "source_key=$DEPLOYMENT_KEY&destination_key=$DEPLOYMENT_KEY&copy_media=1&from_backup=1" "https://flightpath.torchbox.com/copy/$HEROKU_APP_NAME_PRODUCTION/$HEROKU_APP_NAME_STAGING/")
echo $JOB_STATUS
export JOB_ID=$(echo $JOB_STATUS | jq -r '.job_id')
echo $JOB_ID

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   561  100   406  100   155    167     64  0:00:02  0:00:02 --:--:--   167
{"time_started":"2022-05-25T05:35:03.936973Z","job_id":"ec2df4ec-ffb6-48fa-aa69-8da838690da7","status":"started","metadata":{"source_app":"rca-production","destination_app":"rca-development","copy_media":true,"create_snapshot":false,"from_backup":true,"source_key":"xxx","destination_key":"xxx"},"log_output":""}
ec2df4ec-ffb6-48fa-aa69-8da838690da7
Success!

[nickmoreton]

The latest changes you made look good.

[victoriachan]

Looks good except for the bit about HerokuAnonymisationAllowedCheck. I suggest removing that. Also the bit about fake email address being too realistic.

Suggestion: Birdbath is using version 0.0.5, I would suggest upgrading it to django-birdbath = "^1.0.0".

[victoriachan]

Looks good except for the bit about HerokuAnonymisationAllowedCheck. I suggest removing that. Also the bit about fake email address being too realistic.

Suggestion: Birdbath is using version 0.0.5, I would suggest upgrading it to django-birdbath = "^1.0.0".

[nickmoreton]

Hi @kevinhowbrook Could you please have a look at the last commits I made here.

It basically reverts earlier Student account anonymisation and brings it in line with our latest plan.

If it's OK I could push it to dev. Thanks

[rachelhsmith]

This is a tricky one!

I'm not sure if it is necessarily a problem but I noticed that running birdbath more than once on dev data throws an error. I got Exception: [IntegrityError('duplicate key value violates unique constraint "auth_group_name_key"\nDETAIL: Key (name)=(Student: joshua.good-0) already exists.\n')]