Daily Check List 13/01/2025 - 17/01/2025

[rosemaryjoconnor]

---

name: Daily Checklist
about: Steps necessary to check the full index and other data loads
title: 'Daily checklist: '
labels: Full Index, Data loads, Assertion Sync and Image Service
assignees: ''

---

Every day

• monitor the index switch

• openstdout in step k. Check index and update collection alias

  • scroll to bottom and record Current#, NEW#, DIFF# values

• all scheduled jobs for the day completed successfully (raise on Slack if not)

• assertions sync has run - check user assertion counts are in DQ Profile using query:
  https://biocache.ala.org.au/occurrences/search?q=

  

• Images processing

  • Images uploads: https://images.ala.org.au/admin/batchUploads
  • batch uploads progressing and not stalled - take note of the lastUpdated field in a loading batch

• DAG runs - ensure no clusters are long-running (ie > 24 hours)

  • In Airflow - select Browse->Dag Runs and filter on running state
    

• clear biocache cache (has to be done after assertion sync and index successfully switched)

  • biocache admin: https://biocache.ala.org.au/admin

Raise any issues on the data management internal channel

• Monday
• Tuesday
• Wednesday
• Thursday
• Friday

Once a week (whenever)

Sensitive data check

• Run SDS script found in: Report-SDS-Information-Prod
• Commit new report to GitHub SDS Information report

Manual Index Switch

If manual index switch is needed run DAG Update-Collection-Alias

Job Schedule

Scheduled jobs - Github config

Daily

Time	Job	Job Name
3am	Index to SOLR
7am	OBIS	dp5183-obis-weekly

Weekly

Day	Time	Job	DR	Job Name
Mon	12am	BIS Weeds	dr27665	dr27665-BISFetcher-weekly
Mon	7am	iNaturalist	dr1411	dr1411-inaturalist-weekly
Mon	7pm	FeralScan	dr19813	dr19813-feralscan-weekly
Tue	12am	Mel AVH	dr13282	dr13282-mel-avh-weekly
Tue	9am	cPlatypus	dr7973	dr7973-cplatypus-weekly
Tue	9am	APII	dr413	dr413-apii-weekly
Tue	9am	Butterflies Aust	dr16457	dr16457-butterflies-weekly
Tue	5pm	Biocollect	dp3903	dp3903-biocollect-weekly
Tue	8pm	Bionet	dr368	dr368-bionet-weekly
Wed	12am	Perth AVH	dr15863	dr15863-perth-avh-weekly
Wed	8am	Questagame	dr1902	dr1902-questagame-weekly
Wed	9am	NatureMapr	dr19123	dr19123-naturemapr-weekly
Wed	1pm	QM	dr344	dr344-QM-weekly
Thu	12am	CANB AVH	dr15860	dr15860-canb-avh-weekly
Thu	12.25am	CNS AVH	dr15867	dr15867-cns-avh-weekly
Thu	12.40am	JCT AVH	dr15868	dr15868-jct-avh-weekly

Monthly

Day	Time	Job	DR	Job Name
First Thu	5.15pm	MV	dr342	dr342-MV-monthly
First Wed	9am	TurtleSat	dr26141	dr26141-turtleSat-monthly
First Wed	9am	UC Genomics	dr25075	dr25075-UCGenomics-monthly
5th	12am	UniMelb AVH	dr13282	dr13282-melu-avh-monthly
5th	12am	Hobart AVH	dr10574	dr10574-ho-avh-monthly
6th	12am	Allan Herbarium	dr27654	dr27654-chr-avh-monthly
6th	12.01am	WELT AVH	dr26642	dr26642-welt-avh-monthly
6th	12.15am	Auckland Museum Botanical	dr26650	dr26650-ak-avh-monthly
6th	12.45am	NZ PDD	dr26651	dr26651-pdd-avh-monthly
19th	12am	NSW AVH	dr15861	dr15861-nsw-avh-monthly
20th	12am	BRI AVH	dr2287	dr2287-bri-avh-monthly
21st	12am	AD AVH	dr15865	dr15865-ad-avh-monthly

[rosemaryjoconnor]

14/01/2025

Daily Checklist

• SOLR index successful

• INFO- There are more records in the new index than the current one. CURRENT#: 148,026,238 NEW#: 148,087,195 DIFF#: 60,957

• INFO- All checks passed successfully. Switching collection...

• Biocache cache cleared.

• Assertions: Exclude records with unresolved user annotations [(10,246 records excluded)]
  (https://biocache.ala.org.au/occurrence/search?q=&qualityProfile=ALA&disableAllQualityFilters=true&fq=userAssertions%3A50001+userAssertions%3A50005)

• iNat images 89.66% complete

• Assertions sync failed

[rosemaryjoconnor]

15/01/2025

Daily Checklist

Successful

• Perth AVH, Questagame, NatureMapr, UC Genomics, TurtleSat
• dr1411 iNat images complete - 423,912

Failed:

• dp5183 - preingestion

ConnectionResetError: [Errno 104] Connection reset by peer
Command exiting with ret '1'

• SOLR index

INFO- There are more records in the new index than the current one. CURRENT#: 148,087,195 NEW#: 148,123,293 
SEVERE- The check_conservation_status failed. Please check the logs for more details.
SEVERE- Some checks failed. Please check the logs for more details.
INFO- Switching collection alias is skipped as some checks failed.

• Assertions Sync yet to run

[rosemaryjoconnor]

16/01/2025

Daily Checklist

• All daily jobs successful
• Index failed as per yesterday, likely to keep occurring for some time
• Index switched successfully
• Assertions sync successful: 10,263 annotation records excluded
• QM images not loading
• SDS Report run and pushed to GH