Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add tox support for testing redshift, bigquery and snowflake #320

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

emmyoop
Copy link
Member

@emmyoop emmyoop commented Oct 18, 2024

Resolves #316

Description & motivation

This PR make a slight update to the current testing layout to allow dbt Labs to start testing this package as part of our own
release process.

This only adds support for testing with snowflake and redshift.

Adding support for Bigquery will involve updating the secrets and can be done in the next round of changes.

To add additional adapter testing support in the future, you will just need to modify

  • .github/workflows/ci_dbt_test_package.yml to pass through the env vars
  • modify profiles.yml to meet the desire var format (example in [dbt-package-testing](https://github.com/dbt-labs/dbt-- package-testing/blob/main/integration_tests/profiles.yml))
  • supported_adapters.env to list the adapters you want to test
  • tox.ini to add the commands for the associated adapter tests

I've left your current testing as is to allow you to decide what to do with it.

Why Do We Need These Changes

As explained in #316, by setting up tox as a standard entry-point, the Core development team at dbt Labs can include this package's integration tests (along with other popular dbt packages) as part of our continuous development & delivery of dbt.

Checklist

  • I have verified that these changes work locally
  • I have updated the README.md (if applicable)
  • I have added an integration test for my fix/feature (if applicable)

Comment on lines 51 to 52
# TODO: below value - does it need to be secret or could we hard code it in tox and here?
REDSHIFT_SPECTRUM_IAM_ROLE: ${{ secrets.REDSHIFT_SPECTRUM_IAM_ROLE }}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we hardcode this? Is it actually a secret?

@emmyoop emmyoop changed the title Add tox support for testing redshift and snowflake Add tox support for testing redshift, bigquery and snowflake Oct 18, 2024
@emmyoop
Copy link
Member Author

emmyoop commented Oct 18, 2024

To get this working I will need the following added as variables instead of secrets:

  1. REDSHIFT_HOST
  2. REDSHIFT_USER
  3. REDSHIFT_PORT
  4. REDSHIFT_DATABASE
  5. SNOWFLAKE_USER
  6. SNOWFLAKE_WAREHOUSE
  7. SNOWFLAKE_ROLE
  8. SNOWFLAKE_DATABASE
  9. BIGQUERY_PROJECT

The following need to be added as secrets

  1. DBT_ENV_SECRET_REDSHIFT_PASS
  2. SNOWFLAKE_ACCOUNT
  3. DBT_ENV_SECRET_SNOWFLAKE_PASS
  4. BIGQUERY_KEYFILE_JSON

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add tox-based Integration Testing support for this package
1 participant