Feature/cdm line item

.quickstart/quickstart.yml

@@ -8,6 +8,8 @@ table_variables:
     - one_time_product
   recharge__charge_tax_line_enabled:
     - charge_tax_line
+  recharge__checkout_enabled:
+    - checkout
 
 destination_configurations:
   databricks:

CHANGELOG.md

@@ -1,3 +1,27 @@
+# dbt_recharge v0.3.0
+[PR #17](https://github.com/fivetran/dbt_recharge/pull/17) includes the following updates:
+
+## Features
+- Introduced the new `recharge__line_item_enhanced` model. This model includes a line item enriched with invoice, subscription, payment, and refund information. This model has been built with the intention of retaining a common line item schema across all other Fivetran billing data models.
+
+## 🚨 Breaking Changes 🚨
+- In the [dbt_recharge_source v0.3.0 release](https://github.com/fivetran/dbt_recharge_source/releases/tag/v0.3.0), the following columns were added to model `stg_recharge__address`:
+  - `country`
+  - `payment_method_id`
+  - ❗Note: If you have already added any of these fields as passthrough columns to the `recharge__address_passthrough_columns` var, you will need to remove or alias these fields from the var to avoid duplicate column errors.
+
+## Additional source package updates ([release notes](https://github.com/fivetran/dbt_recharge_source/releases/tag/v0.3.0))
+- Added staging model `stg_recharge__checkout`. See [this doc](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge_source.stg_recharge__checkout) for the fields added and their definitions.
+  - This model is disabled by default but can be enabled by setting variable `recharge__checkout_enabled` to true in your `dbt_project.yml` file. See the [Enable/disable models and sources section](https://github.com/fivetran/dbt_recharge/blob/main/README.md#step-4-enable-disable-models-and-sources) of the README for more information.
+  - This model can also be passed additional columns beyond the predefined columns by using the variable `recharge__checkout_passthrough_columns`. See the [Passing Through Additional Columns](https://github.com/fivetran/dbt_recharge/blob/main/README.md#passing-through-additional-columns) section of the README for more information on how to set this variable.
+
+- Added the following columns to model `stg_recharge__customer`. See [this doc](https://fivetran.github.io/dbt_recharge/#!/model/model.recharge_source.stg_recharge__customer) for field definitions.
+  - `billing_first_name`
+  - `billing_last_name`
+  - `billing_company`
+  - `billing_city`
+  - `billing_country`
+
 # dbt_recharge v0.2.0
 [PR #16](https://github.com/fivetran/dbt_recharge/pull/16) includes the following updates:
 ## Features

README.md

@@ -44,7 +44,7 @@ Include the following recharge package version in your `packages.yml` file.
 ```yaml
 packages:
   - package: fivetran/recharge
-    version: [">=0.2.0", "<0.3.0"] # we recommend using ranges to capture non-breaking changes automatically
+    version: [">=0.3.0", "<0.4.0"] # we recommend using ranges to capture non-breaking changes automatically
 ```
 Do **NOT** include the `recharge_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 
@@ -65,17 +65,18 @@ vars:
   recharge_schema: your_schema_name 
 ```
 
-## Step 4: Disable models for non-existent sources
+## Step 4: Enable/disable models and sources
 Your Recharge connector may not sync every table that this package expects. If you do not have the `ONE_TIME_PRODUCT` and/or `CHARGE_TAX_LINE` tables synced, add the corresponding variable(s) to your root `dbt_project.yml` file to disable these sources:
 
 ```yml
 vars:
   recharge__one_time_product_enabled: false # Disables if you do not have the ONE_TIME_PRODUCT table. Default is True.
   recharge__charge_tax_line_enabled: false # Disables if you do not have the CHARGE_TAX_LINE table. Default is True.
-``` 
+  recharge__checkout_enabled: true # Enables if you do have the CHECKOUT table. Default is False.
+```
 
 ## (Optional) Step 5: Additional configurations
-<details open><summary>Expand for configurations</summary>
+<details open><summary>Expand/collapse section.</summary>
 
 ### Leveraging `orders` vs `order` source
 For Fivetran Recharge connectors created on or after June 18, 2024, the `ORDER` source table has been renamed to `ORDERS`. Refer to the [June 2024 connector release notes](https://fivetran.com/docs/connectors/applications/recharge/changelog#june2024) for more information.
@@ -105,7 +106,8 @@ vars:
         transform_sql:  "cast(a_second_field as string)"
     # a similar pattern can be applied to the rest of the following variables.
     recharge__charge_passthrough_columns:
-    recharge__charge_line_item_passthrough_columns: 
+    recharge__charge_line_item_passthrough_columns:
+    recharge__checkout_passthrough_columns:
     recharge__order_passthrough_columns:
     recharge__order_line_passthrough_columns:
     recharge__subscription_passthrough_columns:
@@ -167,7 +169,7 @@ This dbt package is dependent on the following dbt packages. Please be aware tha
 ```yml
 packages:
     - package: fivetran/recharge_source
-      version: [">=0.2.0", "<0.3.0"]
+      version: [">=0.3.0", "<0.4.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,6 +1,6 @@
 
 name: 'recharge'
-version: '0.2.0'
+version: '0.3.0'
 config-version: 2
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 
@@ -15,6 +15,7 @@ vars:
     charge_order_attribute: "{{ ref('stg_recharge__charge_order_attribute') }}"
     charge_tax_line: "{{ ref('stg_recharge__charge_tax_line') }}"
     charge_shipping_line: "{{ ref('stg_recharge__charge_shipping_line') }}"
+    checkout: "{{ ref('stg_recharge__checkout') }}"
     customer: "{{ ref('stg_recharge__customer') }}"
     discount: "{{ ref('stg_recharge__discount') }}"
     one_time_product: "{{ ref('stg_recharge__one_time_product') }}"
@@ -26,7 +27,8 @@ vars:
 
     recharge__address_passthrough_columns: [] 
     recharge__charge_passthrough_columns: [] 
-    recharge__charge_line_item_passthrough_columns: [] 
+    recharge__charge_line_item_passthrough_columns: []
+    recharge__checkout_passthrough_columns: [] 
     recharge__order_passthrough_columns: [] 
     recharge__order_line_passthrough_columns: [] 
     recharge__subscription_passthrough_columns: [] 

integration_tests/dbt_project.yml

@@ -1,11 +1,12 @@
 config-version: 2
 
 name: 'recharge_integration_tests'
-version: '0.2.0'
+version: '0.3.0'
 
 profile: 'integration_tests' 
 
 vars:
+  # recharge__checkout_enabled: true # Uncomment for generating docs -- Comment this out before sending to buildkite.
   recharge_source:
     recharge_schema: recharge_integration_tests_03
     recharge_address_identifier: "address_data"
@@ -17,6 +18,7 @@ vars:
     recharge_charge_order_attribute_identifier: "charge_order_attribute_data"
     recharge_charge_shipping_line_identifier: "charge_shipping_line_data"
     recharge_charge_tax_line_identifier: "charge_tax_line_data"
+    recharge_checkout_identifier: "checkout_data"
     recharge_customer_identifier: "customer_data"
     recharge_discount_identifier: "discount_data"
     recharge_one_time_product_identifier: "one_time_product_data"
@@ -52,13 +54,17 @@ seeds:
       EXTERNAL_PRODUCT_ID_ECOMMERCE: "{{ 'INT64' if target.type == 'bigquery' else 'bigint' }}"
       EXTERNAL_VARIANT_ID_ECOMMERCE: "{{ 'INT64' if target.type == 'bigquery' else 'bigint' }}"
       SUBSCRIPTION_ID: "{{ 'INT64' if target.type == 'bigquery' else 'bigint' }}"
+      COMPANY: "{{ 'string' if target.name in ['bigquery', 'spark', 'databricks'] else 'varchar' }}"
     charge_data:
       +column_types:
         LAST_CHARGE_ATTEMPT_DATE: "timestamp"
         RETRY_DATE: "timestamp"
     charge_discount_data:
       +column_types:
         VALUE: "float"
+    checkout_data:
+      +column_types:
+        COMPLETED_AT: "timestamp"
     customer_data:
       +column_types:
         FIRST_CHARGE_PROCESSED_AT: "timestamp"

integration_tests/seeds/checkout_data.csv

@@ -0,0 +1,2 @@
+TOKEN,CHARGE_ID,BUYER_ACCEPTS_MARKETING,COMPLETED_AT,CREATED_AT,CURRENCY,DISCOUNT_CODE,EMAIL,EXTERNAL_CHECKOUT_ID,EXTERNAL_CHECKOUT_SOURCE,EXTERNAL_TRANSACTION_ID_PAYMENT_PROCESSOR,ORDER_ATTRIBUTES,PHONE,REQUIRES_SHIPPING,SUBTOTAL_PRICE,TAXES_INCLUDED,TOTAL_PRICE,TOTAL_TAX,UPDATED_AT
+abc,400000001,true,,,USD,,,,,,,,,,,,,

models/common_data_models/recharge__common_data_models.yml

@@ -0,0 +1,79 @@
+version: 2
+
+models:
+  - name: recharge__line_item_enhanced
+    description: This table is a standardized model designed to consolidate and enhance line item data and charges. It captures detailed information about each line item, including product details, payment methods, subscription information, customer details, and various financial metrics like taxes, discounts, and total amounts. This unified model facilitates comprehensive analysis and reporting across different transactional data sources.
+    tests:
+      - dbt_utils.unique_combination_of_columns:
+          combination_of_columns:
+            - header_id
+            - line_item_index
+    columns:
+      - name: header_id
+        description: "ID of charge."
+      - name: line_item_id
+        description: "ID of either the charge line item."
+      - name: line_item_index
+        description: "Numerical identifier of the line item within the header object."
+      - name: record_type
+        description: "Either 'header' or 'line_item' to differentiate if the record is originally from the line item table or was created to document information only available at the header level."
+      - name: created_at
+        description: "Date the line item entry was created."
+      - name: header_status
+        description: "Indicates the status of the header. Eg. paid, voided, returned."
+      - name: billing_type
+        description: "Billing type of the charge (e.g., checkout, recurring)."
+      - name: currency
+        description: "Determines the currency in which the checkout took place."
+      - name: product_id
+        description: "ID of the product associated with the line item."
+      - name: product_name
+        description: "Name of the product associated with the line item."
+      - name: product_type
+        description: "Type of the product (e.g., physical, digital)."
+      - name: transaction_type
+        description: "Transaction type of the line item (e.g., subscription, onetime)."
+      - name: quantity
+        description: "Quantity of the product in the line item."
+      - name: unit_amount
+        description: "Unit price of the product."
+      - name: discount_amount
+        description: "Amount of discount applied to the charge."
+      - name: tax_amount
+        description: "Amount of tax applied to the line item."
+      - name: total_amount
+        description: "Total amount for the line item (including tax and discounts)."
+      - name: payment_id
+        description: "ID of the payment associated with the line item."
+      - name: payment_method
+        description: "Method used for payment (e.g., credit card, PayPal)."
+      - name: payment_method_id
+        description: "ID of the payment method."
+      - name: payment_at
+        description: "Date and time of the payment."
+      - name: refund_amount
+        description: "Amount refunded for the charge."
+      - name: subscription_id
+        description: "ID of the subscription (if applicable)."
+      - name: subscription_period_started_at
+        description: "Start date of the subscription period (if applicable)."
+      - name: subscription_period_ended_at
+        description: "End date of the subscription period (if applicable)."
+      - name: subscription_status
+        description: "Status of the subscription."
+      - name: customer_id
+        description: "ID of the customer"
+      - name: customer_level
+        description: "Identifies if the customer is reported at the account or customer level."
+      - name: customer_name
+        description: "Name of the customer."
+      - name: customer_company
+        description: "Company name of the customer."
+      - name: customer_email
+        description: "Email address of the customer."
+      - name: customer_city
+        description: "City of the customer's address."
+      - name: customer_country
+        description: "Country of the customer's address."
+      - name: fee_amount
+        description: "Shipping fees associated with the charge."