bugfix/klaviyo-too-many-partitions

[fivetran-joemarkiewicz]

PR Overview

This PR will address the following Issue/Feature: v0.7.0

This PR will result in the following new package version: No linked issue as this is a result of an upstream change in the Klaviyo package. See dbt_klaviyo PR 41 for more details.

This is a breaking change due to the upstream changes which require a full refresh for bigquery users.

Please provide the finalized CHANGELOG entry which details the relevant changes included in this PR:

Upstream Klaviyo Breaking Changes (Full refresh required after upgrading)

• Upstream incremental models within the dbt_klaviyo package running on BigQuery have had the partition_by logic removed from incremental models running on BigQuery. This change affects only BigQuery warehouses and resolves the too many partitions error that some users encountered. The partitioning was also deemed unnecessary for the mentioned models and their downstream references, offering no performance benefit. By removing it, we eliminate both the error risk and an unneeded configuration. Refer to the v0.8.0 dbt_klaviyo release notes for more details. This change applies to the following models:
  • int_klaviyo__event_attribution
  • klaviyo__events

Under the Hood

• Added consistency and integrity validation tests for the following models:
  • shopify_holistic_reporting__customer_enhanced
  • shopify_holistic_reporting__daily_customer_metrics
  • shopify_holistic_reporting__orders_attribution
• Cleaned up unnecessary variable configuration within the integration_tests/dbt_project.yml file.

PR Checklist

Basic Validation

Please acknowledge that you have successfully performed the following commands locally:

• dbt run –full-refresh && dbt test
• dbt run (if incremental models are present) && dbt test

Before marking this PR as "ready for review" the following have been applied:

• The appropriate issue has been linked, tagged, and properly assigned
• All necessary documentation and version upgrades have been applied
• docs were regenerated (unless this PR does not include any code or yml updates)
• BuildKite integration tests are passing
• Detailed validation steps have been provided below

Detailed Validation

Please share any and all of your validation steps:

See below for the results of the validation tests (full refresh and incremental runs)

If you had to summarize this PR in an emoji, which would it be?

🐟

[fivetran-catfritz]

One small change but the rest lgtm!

[fivetran-avinash]

@fivetran-joemarkiewicz A few small CHANGELOG revisions.

Before approving, I also noticed that shopify_holistic_reporting__orders_attribution (link) had similar partition by logic that was removed in dbt_klaviyo. It seems created_timestamp is involved in a join in that model, so it might hold more value than the partitions we've removed in dbt_klaviyo, but I'm wondering if the config needs to be modified here to improve performance. Curious as to your thoughts here!

[fivetran-joemarkiewicz]

@fivetran-avinash good callout on the partition present in the orders attribution model. After looking into it, I see we are aggressively partitioning on the timestamp grain for a field we aren't including in our incremental logic. Therefore, removing the partition would be more appropriate in this case. See the updated validations on the latest commit.

[fivetran-avinash]

@fivetran-joemarkiewicz Thanks for the quick turnaround! One small change, but otherwise good to go. Make sure to update the packages!