Feature/cdm line item

.quickstart/quickstart.yml

@@ -8,6 +8,8 @@ table_variables:
     - one_time_product
   recharge__charge_tax_line_enabled:
     - charge_tax_line
+  recharge__checkout_enabled:
+    - checkout
 
 destination_configurations:
   databricks:

CHANGELOG.md

@@ -1,3 +1,27 @@
+# dbt_recharge v0.3.0
+[PR #17](https://github.com/fivetran/dbt_recharge/pull/17) includes the following updates:
+
+## Features
+- Introduced the new `recharge__line_item_enhanced` model. This model includes a line item enriched with invoice, subscription, payment, and refund information. This model has been built with the intention of retaining a common line item schema across all other Fivetran billing data models.
+
+## 🚨 Breaking Changes 🚨
+- In the [dbt_recharge_source v0.3.0 release](https://github.com/fivetran/dbt_recharge_source/releases/tag/v0.3.0), the following columns were added to model `stg_recharge__address`:
+  - `country`
+  - `payment_method_id`
+  - ❗Note: If you have already added any of these fields as passthrough columns to the `recharge__address_passthrough_columns` var, you will need to remove or alias these fields from the var to avoid duplicate column errors.
+
+## Additional source package updates ([release notes](https://github.com/fivetran/dbt_recharge_source/releases/tag/v0.3.0))
+- Added staging model `stg_recharge__checkout`. See [this doc](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge_source.stg_recharge__checkout) for the fields added and their definitions.
+  - This model is disabled by default but can be enabled by setting variable `recharge__checkout_enabled` to true in your `dbt_project.yml` file. See the [Enable/disable models and sources section](https://github.com/fivetran/dbt_recharge/blob/main/README.md#step-4-enable-disable-models-and-sources) of the README for more information.
+  - This model can also be passed additional columns beyond the predefined columns by using the variable `recharge__checkout_passthrough_columns`. See the [Passing Through Additional Columns](https://github.com/fivetran/dbt_recharge/blob/main/README.md#passing-through-additional-columns) section of the README for more information on how to set this variable.
+
+- Added the following columns to model `stg_recharge__customer`. See [this doc](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge_source.stg_recharge__customer) for field definitions.
+  - `billing_first_name`
+  - `billing_last_name`
+  - `billing_company`
+  - `billing_city`
+  - `billing_country`
+
 # dbt_recharge v0.2.0
 [PR #16](https://github.com/fivetran/dbt_recharge/pull/16) includes the following updates:
 ## Features

README.md

@@ -44,7 +44,7 @@ Include the following recharge package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/recharge
-    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 Do **NOT** include the `recharge_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 
@@ -65,17 +65,18 @@ vars:
   recharge_schema: your_schema_name 
 ```
 
-## Step 4: Disable models for non-existent sources
+## Step 4: Enable/disable models and sources
 Your Recharge connector may not sync every table that this package expects. If you do not have the `ONE_TIME_PRODUCT` and/or `CHARGE_TAX_LINE` tables synced, add the corresponding variable(s) to your root `dbt_project.yml` file to disable these sources:
 
 ```yml
 vars:
   recharge__one_time_product_enabled: false # Disables if you do not have the ONE_TIME_PRODUCT table. Default is True.
   recharge__charge_tax_line_enabled: false # Disables if you do not have the CHARGE_TAX_LINE table. Default is True.
-``` 
+  recharge__checkout_enabled: true # Enables if you do have the CHECKOUT table. Default is False.
+```
 
 ## (Optional) Step 5: Additional configurations
-<details open><summary>Expand for configurations</summary>
+<details open><summary>Expand/collapse section.</summary>
 
 ### Leveraging `orders` vs `order` source
 For Fivetran Recharge connectors created on or after June 18, 2024, the `ORDER` source table has been renamed to `ORDERS`. Refer to the [June 2024 connector release notes](https://fivetran.com/docs/connectors/applications/recharge/changelog#june2024) for more information.
@@ -105,7 +106,8 @@ vars:
         transform_sql:  "cast(a_second_field as string)"
     # a similar pattern can be applied to the rest of the following variables.
     recharge__charge_passthrough_columns:
-    recharge__charge_line_item_passthrough_columns: 
+    recharge__charge_line_item_passthrough_columns:
+    recharge__checkout_passthrough_columns:
     recharge__order_passthrough_columns:
     recharge__order_line_passthrough_columns:
     recharge__subscription_passthrough_columns:
@@ -167,7 +169,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/recharge_source
-      version: [">=0.2.0", "<0.3.0"]
+      version: [">=0.3.0", "<0.4.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,6 +1,6 @@
 
 name: 'recharge'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
@@ -15,6 +15,7 @@ vars:
     charge_order_attribute: "{{ ref('stg_recharge__charge_order_attribute') }}"
     charge_tax_line: "{{ ref('stg_recharge__charge_tax_line') }}"
     charge_shipping_line: "{{ ref('stg_recharge__charge_shipping_line') }}"
+    checkout: "{{ ref('stg_recharge__checkout') }}"
     customer: "{{ ref('stg_recharge__customer') }}"
     discount: "{{ ref('stg_recharge__discount') }}"
     one_time_product: "{{ ref('stg_recharge__one_time_product') }}"
@@ -26,7 +27,8 @@ vars:
 
     recharge__address_passthrough_columns: [] 
     recharge__charge_passthrough_columns: [] 
-    recharge__charge_line_item_passthrough_columns: [] 
+    recharge__charge_line_item_passthrough_columns: []
+    recharge__checkout_passthrough_columns: [] 
     recharge__order_passthrough_columns: [] 
     recharge__order_line_passthrough_columns: [] 
     recharge__subscription_passthrough_columns: [] 

integration_tests/dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 
 name: 'recharge_integration_tests'
-version: '0.2.0'
+version: '0.3.0'
 
 profile: 'integration_tests' 
 
@@ -17,6 +17,7 @@ vars:
     recharge_charge_order_attribute_identifier: "charge_order_attribute_data"
     recharge_charge_shipping_line_identifier: "charge_shipping_line_data"
     recharge_charge_tax_line_identifier: "charge_tax_line_data"
+    recharge_checkout_identifier: "checkout_data"
     recharge_customer_identifier: "customer_data"
     recharge_discount_identifier: "discount_data"
     recharge_one_time_product_identifier: "one_time_product_data"
@@ -52,13 +53,17 @@ seeds:
       EXTERNAL_PRODUCT_ID_ECOMMERCE: "{{ 'INT64' if target.type == 'bigquery' else 'bigint' }}"
       EXTERNAL_VARIANT_ID_ECOMMERCE: "{{ 'INT64' if target.type == 'bigquery' else 'bigint' }}"
       SUBSCRIPTION_ID: "{{ 'INT64' if target.type == 'bigquery' else 'bigint' }}"
+      COMPANY: "{{ 'string' if target.name in ['bigquery', 'spark', 'databricks'] else 'varchar' }}"
     charge_data:
       +column_types:
         LAST_CHARGE_ATTEMPT_DATE: "timestamp"
         RETRY_DATE: "timestamp"
     charge_discount_data:
       +column_types:
         VALUE: "float"
+    checkout_data:
+      +column_types:
+        COMPLETED_AT: "timestamp"
     customer_data:
       +column_types:
         FIRST_CHARGE_PROCESSED_AT: "timestamp"

integration_tests/seeds/checkout_data.csv

@@ -0,0 +1,2 @@
+TOKEN,CHARGE_ID,BUYER_ACCEPTS_MARKETING,COMPLETED_AT,CREATED_AT,CURRENCY,DISCOUNT_CODE,EMAIL,EXTERNAL_CHECKOUT_ID,EXTERNAL_CHECKOUT_SOURCE,EXTERNAL_TRANSACTION_ID_PAYMENT_PROCESSOR,ORDER_ATTRIBUTES,PHONE,REQUIRES_SHIPPING,SUBTOTAL_PRICE,TAXES_INCLUDED,TOTAL_PRICE,TOTAL_TAX,UPDATED_AT
+abc,400000001,true,,,USD,,,,,,,,,,,,,

models/common_data_models/docs.md

@@ -0,0 +1,127 @@
+{% docs billing_type %}
+Billing type of the charge.
+{% enddocs %}
+
+{% docs created_at %}
+Date the line item entry was created.
+{% enddocs %}
+
+{% docs currency %}
+Determines the currency in which the transaction took place.
+{% enddocs %}
+
+{% docs customer_city %}
+City of the customer's address.
+{% enddocs %}
+
+{% docs customer_company %}
+Company name of the customer (if applicable).
+{% enddocs %}
+
+{% docs customer_country %}
+Country of the customer's address.
+{% enddocs %}
+
+{% docs customer_email %}
+Email address of the customer.
+{% enddocs %}
+
+{% docs customer_level %}
+Identifies if the customer is reported at the account or customer level.
+{% enddocs %}
+
+{% docs customer_name %}
+Name of the customer.
+{% enddocs %}
+
+{% docs discount_amount %}
+Amount of discount applied to the line item.
+{% enddocs %}
+
+{% docs fee_amount %}
+Any additional fees associated with the payment.
+{% enddocs %}
+
+{% docs header_id %}
+ID of either the invoice or order.
+{% enddocs %}
+
+{% docs header_status %}
+Indicates the status of the header. Eg. paid, voided, returned.
+{% enddocs %}
+
+{% docs line_item_id %}
+ID of either the invoice line or order line item.
+{% enddocs %}
+
+{% docs line_item_index %}
+Numerical identifier of the line item within the header object.
+{% enddocs %}
+
+{% docs payment_at %}
+Date and time of the payment.
+{% enddocs %}
+
+{% docs payment_id %}
+ID of the payment associated with the line item.
+{% enddocs %}
+
+{% docs payment_method %}
+Method used for payment (e.g., credit card, PayPal).
+{% enddocs %}
+
+{% docs payment_method_id %}
+ID of the payment method.
+{% enddocs %}
+
+{% docs product_category %}
+Category to which the product belongs (e.g., electronics, clothing).
+{% enddocs %}
+
+{% docs product_id %}
+ID of the product associated with the line item.
+{% enddocs %}
+
+{% docs product_name %}
+Name of the product associated with the line item.
+{% enddocs %}
+
+{% docs product_type %}
+Type of the product (e.g., physical, digital).
+{% enddocs %}
+
+{% docs quantity %}
+Quantity of the product in the line item.
+{% enddocs %}
+
+{% docs record_type %}
+Either 'header' or 'line_item' to differentiate if the record is originally from the line item table or was created to document information only available at the header level.
+{% enddocs %}
+
+{% docs refund_amount %}
+Amount refunded for the line item (if applicable).
+{% enddocs %}
+
+{% docs subscription_period_ended_at %}
+End date of the subscription period (if applicable).
+{% enddocs %}
+
+{% docs subscription_period_started_at %}
+Start date of the subscription period (if applicable).
+{% enddocs %}
+
+{% docs tax_amount %}
+Amount of tax applied to the line item.
+{% enddocs %}
+
+{% docs total_amount %}
+Total amount for the line item (including tax and discounts).
+{% enddocs %}
+
+{% docs transaction_type %}
+Type of the line item.
+{% enddocs %}
+
+{% docs unit_amount %}
+Unit price of the product.
+{% enddocs %}

models/common_data_models/recharge__common_data_models.yml

@@ -0,0 +1,81 @@
+version: 2
+
+models:
+  - name: recharge__line_item_enhanced
+    description: Add description
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - header_id
+            - line_item_index
+    columns:
+      - name: header_id
+        description: "{{ doc('header_id') }}"
+      - name: line_item_id
+        description: "{{ doc('line_item_id') }}"
+      - name: line_item_index
+        description: "{{ doc('line_item_index') }}"
+      - name: record_type
+        description: "{{ doc('record_type') }}"
+      - name: created_at
+        description: "{{ doc('created_at') }}"
+      - name: header_status
+        description: "{{ doc('header_status') }}"
+      - name: billing_type
+        description: "{{ doc('billing_type') }}"
+      - name: currency
+        description: "{{ doc('currency') }}"
+      - name: product_id
+        description: "{{ doc('product_id') }}"
+      - name: product_name
+        description: "{{ doc('product_name') }}"
+      - name: product_type
+        description: "{{ doc('product_type') }}"
+      - name: transaction_type
+        description: "{{ doc('transaction_type') }}"
+      - name: product_category
+        description: "{{ doc('product_category') }}"
+      - name: quantity
+        description: "{{ doc('quantity') }}"
+      - name: unit_amount
+        description: "{{ doc('unit_amount') }}"
+      - name: discount_amount
+        description: "{{ doc('discount_amount') }}"
+      - name: tax_amount
+        description: "{{ doc('tax_amount') }}"
+      - name: total_amount
+        description: "{{ doc('total_amount') }}"
+      - name: payment_id
+        description: "{{ doc('payment_id') }}"
+      - name: payment_method
+        description: "{{ doc('payment_method') }}"
+      - name: payment_method_id
+        description: "{{ doc('payment_method_id') }}"
+      - name: payment_at
+        description: "{{ doc('payment_at') }}"
+      - name: refund_amount
+        description: "{{ doc('refund_amount') }}"
+      - name: subscription_id
+        description: "{{ doc('subscription_id') }}"
+      - name: subscription_period_started_at
+        description: "{{ doc('subscription_period_started_at') }}"
+      - name: subscription_period_ended_at
+        description: "{{ doc('subscription_period_ended_at') }}"
+      - name: subscription_status
+        description: "{{ doc('subscription_status') }}"
+      - name: customer_id
+        description: "{{ doc('customer_id') }}"
+      - name: customer_level
+        description: "{{ doc('customer_level') }}"
+      - name: customer_name
+        description: "{{ doc('customer_name') }}"
+      - name: customer_company
+        description: "{{ doc('customer_company') }}"
+      - name: customer_email
+        description: "{{ doc('customer_email') }}"
+      - name: customer_city
+        description: "{{ doc('customer_city') }}"
+      - name: customer_country
+        description: "{{ doc('customer_country') }}"
+      - name: fee_amount
+        description: "{{ doc('fee_amount') }}"