Merge pull request #9 from fivetran/bugfix/updated-orders

use updated_timestamp instead of created_timestamp in incremental logic
fivetran · Oct 13, 2022 · 037a882 · 037a882
2 parents 08ac66f + 2d872c0
commit 037a882
Show file tree

Hide file tree

Showing 16 changed files with 48 additions and 38 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -22,17 +22,6 @@ jobs:
             pip install -r integration_tests/requirements.txt
             mkdir -p ~/.dbt
             cp integration_tests/ci/sample.profiles.yml ~/.dbt/profiles.yml
-      - run:
-          name: "Run Tests - Postgres"
-          command: |
-            . venv/bin/activate
-            echo `pwd`
-            cd integration_tests
-            dbt deps
-            dbt seed --target postgres --full-refresh
-            dbt run --target postgres --full-refresh
-            dbt run --target postgres
-            dbt test --target postgres
       - run:
           name: "Run Tests - BigQuery"
           environment:
@@ -47,6 +36,17 @@ jobs:
             dbt run --target bigquery --full-refresh
             dbt run --target bigquery
             dbt test --target bigquery
+      - run:
+          name: "Run Tests - Postgres"
+          command: |
+            . venv/bin/activate
+            echo `pwd`
+            cd integration_tests
+            dbt deps
+            dbt seed --target postgres --full-refresh
+            dbt run --target postgres --full-refresh
+            dbt run --target postgres
+            dbt test --target postgres
       - run:
           name: "Run Tests - Redshift"
           command: |

diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md
@@ -1,4 +1,3 @@
-Pull Request
 **Are you a current Fivetran customer?** 
 <!--- Please tell us your name, title and company -->
 

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -1,3 +1,12 @@
+# dbt_shopify_holistic_reporting v0.2.0
+## Bug Fixes
+- Adjusts the incremental logic in the `shopify_holistic_reporting__orders_attribution` model. Previously, on incremental runs, this model transformed only newly-_created_ orders, comparing each order's `created_timestamp` to the `max(created_timestamp)` in the model. Now, the model will also transform newly-_updated_ orders and use `updated_timestamp` instead of `created_timestamp` to determine if an order should be included in an incremental run ([#9](https://github.com/fivetran/dbt_shopify_holistic_reporting/pull/9)).
+
+This is a 🚨 Breaking Change 🚨 as **you will need to run a full refresh**.
+
+## Under the Hood
+- Ensures that the incremental strategy used by Postgres and Redshift adapters in the `shopify_holistic_reporting__orders_attribution` model is `delete+insert` ([#9](https://github.com/fivetran/dbt_shopify_holistic_reporting/pull/9)). Newer versions of dbt introduced an error message if the provided incremental strategy is not `append` or `delete+insert` for these adapters.
+
 # dbt_shopify_holistic_reporting v0.1.1
 ## Bug Fixes
 Incorporate the try_cast macro from fivetran_utils to ensure that the numeric_value field in int__daily_klaviyo_user_metrics is the same data type as '0'. ([#6](https://github.com/fivetran/dbt_shopify_holistic_reporting/pull/6))

diff --git a/README.md b/README.md
@@ -16,7 +16,7 @@ This package produces three final output models, and is currently designed to wo
 
 | **model**                | **description**                                                                                                                                |
 | ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------- |
-| [shopify_holistic_reporting__orders_attribution](https://github.com/fivetran/dbt_shopify_holistic_reporting/blob/main/models/shopify_holistic_reporting__orders_attribution.sql)             | Each record represents a unique Shopify order, enhanced with a customizable last-touch attribution model associating orders with Klaviyo flows and campaigns that customers interacted with. Includes dimensions like whether it is a new or repeat purchase in Shopify. See available customizations [here](https://github.com/fivetran/dbt_klaviyo#attribution-lookback-window). |
+| [shopify_holistic_reporting__orders_attribution](https://github.com/fivetran/dbt_shopify_holistic_reporting/blob/main/models/shopify_holistic_reporting__orders_attribution.sql)             | Each record represents a unique Shopify order, enhanced with a customizable last-touch attribution model associating orders with Klaviyo flows and campaigns that customers interacted with. Includes dimensions like whether it is a new or repeat purchase in Shopify. See available customizations [here](https://github.com/fivetran/dbt_klaviyo#attribution-lookback-window). Materialized incrementally by default. |
 | [shopify_holistic_reporting__daily_customer_metrics](https://github.com/fivetran/dbt_shopify_holistic_reporting/blob/main/models/shopify_holistic_reporting__daily_customer_metrics.sql)             | Each record represent a unique customer's daily activity attributed to a campaign or flow in Klaviyo. The grain is set at the customer-day-flow/campaign level. This model is enriched with both Shopify and Klaviyo metrics, such as the net revenue, taxes paid, discounts applied, and the counts of each type of interaction between the user and the campaign/flow. |
 | [shopify_holistic_reporting__customer_enhanced](https://github.com/fivetran/dbt_shopify_holistic_reporting/blob/main/models/shopify_holistic_reporting__customer_enhanced.sql)             | Each record represents a unique individual (based on email) that may exist in Shopify, Klaviyo, or both platforms. Enhanced with information coalesced across platforms, lifetime order metrics, and all-time interactions with email marketing campaigns and flows. |
 

diff --git a/dbt_project.yml b/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'shopify_holistic_reporting'
-version: '0.1.1'
+version: '0.2.0'
 config-version: 2
 
 require-dbt-version: [">=1.0.0", "<2.0.0"]

diff --git a/docs/catalog.json b/docs/catalog.json
diff --git a/docs/graph.gpickle b/docs/graph.gpickle
diff --git a/docs/index.html b/docs/index.html
diff --git a/docs/manifest.json b/docs/manifest.json
diff --git a/docs/partial_parse.msgpack b/docs/partial_parse.msgpack
diff --git a/docs/run_results.json b/docs/run_results.json
diff --git a/integration_tests/dbt_project.yml b/integration_tests/dbt_project.yml
@@ -1,5 +1,5 @@
 name: 'shopify_holistic_reporting_integration_tests'
-version: '0.1.1'
+version: '0.2.0'
 profile: 'integration_tests'
 config-version: 2
 

diff --git a/integration_tests/requirements.txt b/integration_tests/requirements.txt
@@ -1,6 +1,6 @@
-dbt-snowflake~=1.0.0
-dbt-bigquery~=1.0.0
-dbt-redshift~=1.0.0
-dbt-postgres~=1.0.0
-dbt-spark~=1.0.0
-dbt-spark[PyHive]~=1.0.0
+dbt-snowflake>=1.0.0,<2.0.0
+dbt-bigquery>=1.0.0,<2.0.0
+dbt-redshift>=1.0.0,<2.0.0
+dbt-postgres>=1.0.0,<2.0.0
+dbt-spark>=1.0.0,<2.0.0
+dbt-spark[PyHive]>=1.0.0,<2.0.0
diff --git a/models/shopify_holistic_reporting.yml b/models/shopify_holistic_reporting.yml
@@ -218,6 +218,8 @@ models:
       `klaviyo__email_attribution_lookback` and `klaviyo__sms_attribution_lookback` variables, respectively (in integer-hours).
 
       Refer to the Klaviyo package [README](https://github.com/fivetran/dbt_klaviyo#attribution-lookback-window) for more details.
+
+      Materialized incrementally by default.
     tests:
       - dbt_utils.unique_combination_of_columns:
           combination_of_columns:

diff --git a/models/shopify_holistic_reporting__orders_attribution.sql b/models/shopify_holistic_reporting__orders_attribution.sql
@@ -6,7 +6,7 @@
             "field": "created_timestamp",
             "data_type": "timestamp"
         } if target.type == 'bigquery' else none,
-        incremental_strategy = 'merge',
+        incremental_strategy = 'merge' if target.type not in ('postgres', 'redshift') else 'delete+insert',
         file_format = 'delta'
     )
 }}
@@ -16,9 +16,9 @@ with orders as (
     select *
     from {{ ref('shopify__orders') }}
 
-    -- just grab new orders
+    -- just grab new + newly updated orders
     {% if is_incremental() %}
-        where created_timestamp >= (select max(created_timestamp) from {{ this }})
+        where updated_timestamp >= (select max(updated_timestamp) from {{ this }})
     {% endif %}
 
 ), events as (

diff --git a/packages.yml b/packages.yml
@@ -3,4 +3,4 @@ packages:
     version: [">=0.6.0", "<0.7.0"]
 
   - package: fivetran/klaviyo
-    version: [">=0.4.0", "<0.5.0"]
+    version: [">=0.4.0", "<0.5.0"]