mixed concerns and dependency management issues

[georgim0]

Hi there,

Excited to see this tutorial as it's something we've been struggling in the past.

I've tried mixing airflow, dbt and ge in the past.. This approach has 2 issues:

• dependency management nightmare. too many transitive dependencies
• mixed concerns:
  • GE performs data quality checks
  • DBT creates/updates tables in your DW
  • Airflow triggers jobs

Here's the approach we've taken:

• have 2 repos:
  • model definitions along with their expectations in 1 repo that spits out 2 dockerised containers:
    • one for dbt runs
    • one for GE runs
  • airflow dag definitions
    • triggers dbt container
    • triggers GE container

Could you validate our approach please? Is GE used the way it was designed to?

[eugmandel]

The approach sounds good to me. You should create one GE DataContext (project) created - probably in the repo with the dbt models.
Then the airflow task that invokes validation loads the project's config from the models repo.
You can parameterize the credentials of the database (your datasource) using env variables.