v2.2.3 Added Badges/Cleaned Readme

Goal:
Cleaner README.md with badges to highlight repo details

PyPi Integration: Fix Remaining Conflicts

Hotfixes for v2.2

V2.2.1 Ensure PyPi and Github Action Run As Expected

Changes:

• Update version in setup.py to v2.2.1
• Update version in action.yml to 2.2.1
• Move relative imports of dbt_table_diff/arg_parser.py from dbt_table_diff/run_sql_checks.py to dbt_table_diff/__init__.py

v2.2 PyPi Package Integration

Goal:

• Add dbt_table_diff as a PyPi package.

---

Use Cases:

1. Code Example:

from dbt_table_diff import run_dbt_table_diff

run_dbt_table_diff(
        project_id="ultimate-bit-359101",
        keyfile_path="secrets/bq_keyfile.json",
        manifest_file="target/manifest.json",
        dev_prefix="dev_",
        prod_prefix="prod_",
        fallback_prefix="fb_",
        custom_checks_path="",
        ignored_schemas=[],
        irregular_schemas=[],
        org_name="org-not-included",
        repo_name="dbt_example",
        pr_id="2",
        auth_token="my_github_pat",
)

---

2. CLI Example:

python3 -m dbt_table_diff -t $GH_TOKEN -o org-not-included -r dbt_example -l 2 \
--manifest_file 'target/manifest.json' --project_id 'ultimate-bit-359101' \
--keyfile_path 'secrets/bq_keyfile.json' --dev_prefix 'dev_' --prod_prefix 'prod_' --fallback_prefix 'fb_'

---

3. (Existing) Github Action Example

v2.2 PyPi Package Integration

Enable dbt_table_diff to be leveraged as:

• Github Action
• CLI Command (PyPi)
• Import Python Package (PyPi)

PyPi Package Integration

v2.2.0-alpha

added github workflow for publishing pypi package

V2.1.1

Fix README.md div headers

V2.1: Documentation

Changes:

• Cleaned up README.md
• Added inline comments to helpers/run_sql_checks.py.

V2: Allow Users to include Custom Tests

Exposes an additional Github Action Input, custom_checks_path, which allows users to specify a local folder that contains custom checks.

The SQL files in custom_checks_path can leverage the following jinja variables (example):

{{ dataset }} --- the BQ project ID
{{ schema }} --- the dev schema
{{ compare_schema }} --- the prod schema
{{ table }} --- the name of the table (not including schema or dataset)

Full Changelog: v1...v2

DBT Row/Col Check on PRs (BQ only)

Description:
This github action is intended for comparing models that have changed in an open PR.

Overview of process:

• Fetches modified files by CURLing github.api.com/repos/{organization}/{repository}/pulls/{pull_request_id}/files
• Filters on files matching models/*.sql (call these relevant_files)
• Runs dbt deps; dbt compile to build manifest.json
• Parses manifest.json and fetches relevant_models with manifest-attribute original_file_path in relevant_files
• Loops over models
  • Runs all SQL files in helpers/sql_checks for each of the relevant_models (comparing dev vs prod via (dev_prefixandprod_prefix`)
  • Saves output to file, in pretty format for github comment
  • Leverages py-github-helper to post comment on open PR

Github Actions Input Arguments:
Look at this working example, if you are unfamiliar with Github Actions Inputs. Inputs are used to configure Github Actions, and can be thought of as parameters/config, so the action knows what to do.

How inputs are used:

• GCP_TOKEN -> for connecting to BQ
  • runs dbt compile and helpers/sql_checks to compare tables
• GH_TOKEN -> for connecting to Github
  • fetches modified models/*.sql in your PR
  • adds comment on your PR
• PR_NUMBER -> for fetching open PR from github (the PR your running this action against)
• GH_REPO -> for fetching open PR from github (the PR your running this action against)
• GH_ORG -> for fetching open PR from github (the PR your running this action against)
• dev_prefix -> the prefix used when running dbt locally (aka your dev environment)
• prod_prefix -> the prefix used when running dbt remotely (aka your prod environment)
• fallback_prefix -> if you have a custom generate_schema_name, where you have a different prefix for some models in prod, you can set this field to that prefix
• project_id -> for connecting to BQ (the BQ project id)
• DBT_PROFILE_FILE -> the local path in your repo to your profile.yml for dbt
  • this is necessary for compiling manifest.json during setup process
• ignored_schemas -> comma separated string of schemas to ignore (skip checking during github action)
• irregular_schemas -> comma separated string of schemas which use fallback_prefix (described above)

Quick Start:
A bare-bones example can be found here.