Merge pull request #61 from fivetran/feature/shopify-api-updates

feature/shopify-api-updates

CHANGELOG.md

@@ -1,3 +1,36 @@
+# dbt_shopify v0.9.0
+([PR #61](https://github.com/fivetran/dbt_shopify/pull/61)) includes the following updates:
+## Breaking Changes
+These changes are made breaking due to changes in the source.
+
+- In [June 2023](https://fivetran.com/docs/applications/shopify/changelog#june2023) the Shopify connector received an update which upgraded the connector to be compatible with the new [2023-04 Shopify API](https://shopify.dev/docs/api). As a result, the following fields have been removed as they were deprecated in the API upgrade: ([dbt_shopify_source PR #70](https://github.com/fivetran/dbt_shopify_source/pull/70))
+
+| **model** | **field removed** |
+|-------|--------------|
+| [stg_shopify__customer](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__customer) | `lifetime_duration` |
+| [stg_shopify__order_line](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order_line) | `fulfillment_service` |
+| [stg_shopify__order_line](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order_line) | `destination_location_*` fields |
+| [stg_shopify__order_line](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order_line) | `origin_location_*` fields |
+| [stg_shopify__order](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order) | `total_price_usd` |
+| [stg_shopify__order](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify_source.stg_shopify__order) | `processing_method` |
+
+- Please be aware that the removal of the fields from the staging models results in the removal of the fields in the relevant downstream models:
+
+| **model** | **field removed** |
+|-------|--------------|
+| [shopify__customer](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__customer) | `lifetime_duration` |
+| [shopify__customer_emails](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__customer_emails) | `lifetime_duration` |
+| [shopify__order_lines](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__order_lines) | `fulfillment_service` |
+| [shopify__order_lines](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__order_lines) | `destination_location_*` fields |
+| [shopify__order_lines](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__order_lines) | `origin_location_*` fields |
+| [shopify__orders](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__orders) | `total_price_usd` |
+| [shopify__orders](https://fivetran.github.io/dbt_shopify/#!/model/model.shopify.shopify__orders) | `processing_method` |
+
+## Documentation Updates
+- The `shopify_using_shop_metafields` variable was added to the Adding Metafields of the README. It was erroneously omitted in a previous release. 
+- Documentation provided in the README for how to connect sources when leveraging the union schema/database feature. 
+- Removal of the `current_total_price` yml definition from the shopify__orders model as it was not being created in the model. 
+
 # dbt_shopify v0.8.1
 
 [PR #58](https://github.com/fivetran/dbt_shopify/pull/58) applies the following changes:

README.md

@@ -54,7 +54,7 @@ If you are **not** using the [Shopify Holistic reporting package](https://github
 ```yml
 packages:
   - package: fivetran/shopify
-    version: [">=0.8.0", "<0.9.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.9.0", "<0.10.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 
 Do **NOT** include the `shopify_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well. 
@@ -88,6 +88,9 @@ vars:
     shopify_union_schemas: ['shopify_usa','shopify_canada'] # use this if the data is in different schemas/datasets of the same database/project
     shopify_union_databases: ['shopify_usa','shopify_canada'] # use this if the data is in different databases/projects but uses the same schema name
 ```
+Please be aware that the native `source.yml` connection set up in the package will not function when the union schema/database feature is utilized. Although the data will be correctly combined, you will not observe the sources linked to the package models in the Directed Acyclic Graph (DAG). This happens because the package includes only one defined `source.yml`.
+
+To connect your multiple schema/database sources to the package models, follow the steps outlined in the [Union Data Defined Sources Configuration](https://github.com/fivetran/dbt_fivetran_utils/tree/releases/v0.4.latest#union_data-source) section of the Fivetran Utils documentation for the union_data macro. This will ensure a proper configuration and correct visualization of connections in the DAG.
 
 ## Step 4: Enable `fulfillment_event` data
 
@@ -202,7 +205,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/shopify_source
-      version: [">=0.8.0", "<0.9.0"]
+      version: [">=0.9.0", "<0.10.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'shopify'
-version: '0.8.1'
+version: '0.9.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:

integration_tests/dbt_project.yml

@@ -1,5 +1,5 @@
 name: 'shopify_integration_tests'
-version: '0.8.1'
+version: '0.9.0'
 profile: 'integration_tests'
 config-version: 2
 
@@ -9,6 +9,7 @@ clean-targets:         # directories to be removed by `dbt clean`
   - "dbt_modules"
 
 vars:
+  shopify_using_fulfillment_event: true
   shopify_schema: shopify_integration_tests_6
   shopify_source:
     shopify_customer_identifier:          "shopify_customer_data"

models/shopify.yml

@@ -153,8 +153,6 @@ models:
         description: The order 's position in the shop's count of orders starting at 1001. Order numbers are sequential and start at 1001.
       - name: processed_timestamp
         description: The date and time when an order was processed. This value is the date that appears on your orders and that's used in the analytic reports.
-      - name: processing_method
-        description: How the payment was processed.
       - name: referring_site
         description: The website where the customer clicked a link to the shop.
       - name: shipping_address_address_1
@@ -274,8 +272,6 @@ models:
         description: The total of all line item prices in shop and presentment currencies.
       - name: total_price_set
         description: The total price of the order in shop and presentment currencies.
-      - name: total_price_usd
-        description: The sum of all line item prices, discounts, shipping, taxes, and tips in USD. Must be positive.
       - name: total_tax_set
         description: The total tax applied to the order in shop and presentment currencies.
       - name: is_confirmed
@@ -288,10 +284,8 @@ models:
         description: The total amount of discount (in shop currency) allocated via a fixed-amount discount
       - name: count_discount_codes_applied
         description: Distinct discount codes applied on the order
-      - name: source_identifier
+      - name: source_relation
         description: The ID of the order placed on the originating platform. This value doesn't correspond to the Shopify ID that's generated from a completed draft.
-      - name: current_total_price
-        description: The current total price of the order in the shop currency. The value of this field reflects order edits, returns, and refunds.
 
   - name: shopify__customers
     description: Each record represents a customer in Shopify.
@@ -379,8 +373,6 @@ models:
         description: >
           Field indicating if the customer has consented to receive marketing material via email. Coalescing of the new `email_marketing_consent_state`
           field and the deprecated `accepts_marketing` field. Records with the old field will be marked with '(legacy)'.
-      - name: lifetime_duration
-        description: The amount of time since the customer was first added to the store.
 
   - name: shopify__customer_emails
     description: Each record represents a customer email in Shopify.
@@ -460,8 +452,6 @@ models:
         description: Average shipping tax per order attributed to the customer.
       - name: marketing_consent_updated_at
         description: Timestamp when marketing consent was updated.
-      - name: lifetime_duration
-        description: The amount of time since the customer was first added to the store.
       - name: currency
         description: The three-letter code (ISO 4217 format) for the currency that the customer used when they paid for their last order. Defaults to the shop currency. Returns the shop currency for test orders.
       - name: marketing_consent_state
@@ -561,8 +551,6 @@ models:
       - name: fulfillable_quantity
         description: >
           The amount available to fulfill, calculated as follows: quantity - max(refunded_quantity, fulfilled_quantity) - pending_fulfilled_quantity - open_fulfilled_quantity
-      - name: fulfillment_service
-        description: The service provider that's fulfilling the item.
       - name: fulfillment_status
         description: How far along an order is in terms line items fulfilled.
       - name: is_gift_card
@@ -667,38 +655,6 @@ models:
         description: The total amount allocated to the line item in the presentment currency.
       - name: properties
         description: Line item properties.
-      - name: destination_location_address_1
-        description: Destination location's street address.
-      - name: destination_location_address_2
-        description: The optional second line of the destination location's street address.
-      - name: destination_location_city
-        description: The city the destination location is in.
-      - name: destination_location_country_code
-        description: The country code of the destination location.
-      - name: destination_location_id
-        description: The ID of the destination location.
-      - name: destination_location_name
-        description: The name of the destination location.
-      - name: destination_location_province_code
-        description: The province, state, or district code (ISO 3166-2 alpha-2 format) of the destination location.
-      - name: destination_location_zip
-        description: The zip or postal code.
-      - name: origin_location_address_1
-        description: Origin location's street address.
-      - name: origin_location_address_2
-        description: The optional second line of the origin location's street address.
-      - name: origin_location_city
-        description: The city the origin location is in.
-      - name: origin_location_country_code
-        description: The country code of the origin location.
-      - name: origin_location_id
-        description: The ID of the origin location.
-      - name: origin_location_name
-        description: The name of the origin location.
-      - name: origin_location_province_code
-        description: The province, state, or district code (ISO 3166-2 alpha-2 format) of the origin location.
-      - name: origin_location_zip
-        description: The zip or postal code.
 
   - name: shopify__transactions
     description: Each record represents a transaction in Shopify.

packages.yml

@@ -1,3 +1,3 @@
 packages:
 - package: fivetran/shopify_source
-  version: [">=0.8.0", "<0.9.0"]
+  version: [">=0.9.0", "<0.10.0"]