Charter 29: Explore QA local database with deleted items to discover if fill in gaps method is working as anticipated

Charter:

• Explore QA local database with deleted items to discover if fill in gaps method is working as anticipated

Testers:

• Mike Walker-Rose

Date:

• 30/08/24

Duration:

• 45 - 90 mins

Scope:

• Local QA database

References:

• Mongo DB Compass

Test Approach:

• run main and establish how many documents are in the last 7 days
• delete specific days and repeat step 1
• explore different days and establish if the data is being filled correctly
• look for re-uploads
• look for things not being overwritten that shouldn't

Notes

• deleted everything from in situ qa db
• at 07:25 UTC ran test__in_situ_etl__calling_actual_api_returns_values_and_stores test from open_aq_etl_test.py which calls main without mocking openaq response, removed the limit for just London to get all results
  • 27453 documents resulted
  • exported to excel, range of documents had measurement dates 2024-08-29T07:26:00.000Z to 2024-08-30T07:14:00.000Z
    • would have expected this to populate with 7 days
• re-run 08:49
  • 27810 documents resulted
  • all within range 29/08 - 30/08
• changed approach, running in situ etl script in case test setup is limiting results - 08:57
  • 216480 documents added
  • 2024-08-30 09:26:20,439 - INFO - 27866 documents upserted, 0 modified
  • 2024-08-23T07:57:09.000Z to 2024-08-30T07:57:00.000Z
• re-run at 10:33
  • 2024-08-30 10:37:30,178 - INFO - 1804 documents upserted, 25368 modified
  • Able to find first and last item from the first dataset and confirmed they still existed and had not been updated
  • later measurements were recorded with a different created_time to the first batch
• delete day from the middle, should be upserted
  • { measurement_date: { $gte: ISODate("2024-08-27T00:00:00.000Z"), $lt: ISODate("2024-08-28T00:00:00.000Z") } }
  • 31300 documents
  • re-run etl, now 220222 documents
  • confirmed first and last documents are present
  • found evidence of data being re-uploaded but with a different created_time before:

after:

Conclusion

• from the data observed this appears to be working as expected