Merge pull request #101 from fivetran/feature/standardize_cent_conversions

move cent conversions upstream to source package

Author: fivetran-reneeli

.buildkite/scripts/run_models.sh

@@ -19,7 +19,7 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{stripe__using_invoices: false, stripe__using_payment_method: false, stripe__using_subscriptions: false, stripe_timezone: "America/New_York", stripe__using_subscription_history: false, stripe__using_price: false}' --target "$db"
+dbt run --vars '{stripe__using_invoices: false, stripe__using_payment_method: false, stripe__using_subscriptions: false, stripe_timezone: "America/New_York", stripe__using_subscription_history: false, stripe__using_price: false, stripe__convert_values: true}' --target "$db"
 dbt test --target "$db"
 
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

CHANGELOG.md

@@ -1,6 +1,28 @@
-# dbt_stripe version.version
+# dbt_stripe v0.16.0
+## Breaking Change - Potential Change in Values May Impact Downstream Queries
+- The aggregated net, gross, sale, charge, refund, and fee amount-based fields in `stripe__daily_overview` and `stripe__customer_overview` now reflect the raw smallest units (e.g., cents), following Stripe's raw data, instead of converted dollar amounts. This change standardizes values across all models.  
+  - Customers using these models should note that the values will appear inflated compared to the previous dollar-based representation.  
+  - The cent-to-dollar conversion has been moved upstream and is now controlled by a new variable: `stripe__convert_values` which is disabled by default.  
+  - See the below Feature Update notes for more context behind these changes and for detailed setup instructions, see the [README](https://github.com/fivetran/dbt_stripe?tab=readme-ov-file#enabling-cent-to-dollar-conversion).  
+
+## Feature Update: Optional Conversion to Major Units
+Stripe passes amount-based fields, such as `amount`, `net`, and `fee`, in the smallest denomination as raw form. This means, if your currency has minor and major units such as USD, 100 represents 100 cents, the minor unit, or 1 USD, the major unit. Alternatively, if your currency doesn't use minor units such as JPY, 100 represents 100 JPY. 
+
+- This PR introduces a variable `stripe__convert_values` (disabled by default) upstream in the staging models which allows users the option to divide all amount-based fields by 100.
+  - For information on how to enable the division, refer to the [README](https://github.com/fivetran/dbt_stripe?tab=readme-ov-file#enabling-cent-to-dollar-conversion) on configuring the `stripe__convert_values` variable.
+  - Otherwise, amount-based fields will be brought through in their raw form.
+  - Examples of currencies using minor units (in which enabling `stripe__convert_values` is relevant) include United States Dollar (USD), Euro (EUR), and the Canadian Dollar (CAD).
+  - Examples of currencies NOT using minor units (in which it makes more sense to keep the amount-based fields in raw form) include Japanese Yen (JPY), Indonesian Rupiah (IDR), and Korean Won (KRW).
+
+## Notes
+- This update is not providing multiple currency support; however, we're interested in exploring this functionality if there's a need. As such, we have created a [feature request to support multiple currencies](https://github.com/fivetran/dbt_stripe/issues/102) where you are welcome to provide feedback or contribute to the discussion.
+
+## Under the Hood
+- Updated the `run_models.sh` script to test for when `stripe__convert_values` is set to True.
+- Added a consistency test for `stripe__customer_overview`.
 
 ## Documentation
+- Updated the descriptions for all amount-based fields to specify the grain of the values and add information about the `stripe__convert_values` variable.
 - Added Quickstart model counts to README. ([#103](https://github.com/fivetran/dbt_stripe/pull/103))
 - Corrected references to connectors and connections in the README. ([#103](https://github.com/fivetran/dbt_stripe/pull/103))
 

README.md

@@ -1,4 +1,6 @@
-<p align="center">
+# Stripe Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_stripe/))
+
+<p align="left">
     <a alt="License"
         href="https://github.com/fivetran/dbt_stripe/blob/main/LICENSE">
         <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
@@ -13,7 +15,6 @@
         <img src="https://img.shields.io/badge/Fivetran_Quickstart_Compatible%3F-yes-green.svg" /></a>
 </p>
 
-# Stripe Transformation dbt Package ([Docs](https://fivetran.github.io/dbt_stripe/))
 ## What does this dbt package do?
 - Produces modeled tables that leverage Stripe data from [Fivetran's connector](https://fivetran.com/docs/applications/stripe) in the format described by [this ERD](https://fivetran.com/docs/applications/stripe#schemainformation) and build off the output of our [stripe source package](https://github.com/fivetran/dbt_stripe_source).
 - Enables you to better understand your Stripe transactions. The package achieves this by performing the following:
@@ -74,7 +75,7 @@ Include the following stripe package version in your `packages.yml` file:
 ```yaml
 packages:
   - package: fivetran/stripe
-    version: [">=0.15.0", "<0.16.0"]
+    version: [">=0.16.0", "<0.17.0"]
 ```
 Do **NOT** include the `stripe_source` package in this file. The transformation package itself has a dependency on it and will install the source package as well.
 
@@ -222,6 +223,17 @@ vars:
     stripe__subscription_metadata: ['the', 'list', 'of', 'property', 'fields'] # Note: this is case-SENSITIVE and must match the casing of the property as it appears in the JSON
 ```
 
+#### Enabling Cent to Dollar Conversion
+
+Amount-based fields, such as `amount` and `net`, are typically displayed in the smallest denomination (e.g., cents for USD). By default, amount-based fields will be in this raw form. However, some currencies use major and minor units (for example, cents and dollars when using USD). In these cases, it may be useful to divide the amounts by 100, converting amounts to major units (dollars for USD). To enable the division, configure the `stripe__convert_values` to `true` in your project.yml: 
+
+```yml
+vars:
+    stripe__convert_values: true  # default is false 
+```
+
+If you are working in a currency that does not differentiate between minor and major units, such as JPY or KRW, it may make more sense to keep the amount-based fields in raw form and therefore the package can be ran without configuration. As `stripe__convert_values` is disabled by default, these fields will not be impacted.
+
 #### Change the build schema
 By default, this package builds the stripe staging models within a schema titled (`<target_schema>` + `_stg_stripe`) in your destination. If this is not where you would like your stripe staging data to be written to, add the following configuration to your root `dbt_project.yml` file:
 
@@ -256,7 +268,7 @@ This dbt package is dependent on the following dbt packages. These dependencies
 ```yml
 packages:
     - package: fivetran/stripe_source
-      version: [">=0.12.0", "<0.13.0"]
+      version: [">=0.13.0", "<0.14.0"]
 
     - package: fivetran/fivetran_utils
       version: [">=0.4.0", "<0.5.0"]

dbt_project.yml

@@ -1,7 +1,7 @@
 config-version: 2
 name: 'stripe'
 
-version: '0.15.1'
+version: '0.16.0'
 require-dbt-version: [">=1.3.0", "<2.0.0"]
 models:
   stripe:

docs/catalog.json

@@ -16,13 +16,13 @@ integration_tests:
       pass: "{{ env_var('CI_REDSHIFT_DBT_PASS') }}"
       dbname: "{{ env_var('CI_REDSHIFT_DBT_DBNAME') }}"
       port: 5439
-      schema: stripe_integrations_tests_15
+      schema: stripe_integrations_tests_19
       threads: 8
     bigquery:
       type: bigquery
       method: service-account-json
       project: 'dbt-package-testing'
-      schema: stripe_integrations_tests_15
+      schema: stripe_integrations_tests_19
       threads: 8
       keyfile_json: "{{ env_var('GCLOUD_SERVICE_KEY') | as_native }}"
     snowflake:
@@ -33,7 +33,7 @@ integration_tests:
       role: "{{ env_var('CI_SNOWFLAKE_DBT_ROLE') }}"
       database: "{{ env_var('CI_SNOWFLAKE_DBT_DATABASE') }}"
       warehouse: "{{ env_var('CI_SNOWFLAKE_DBT_WAREHOUSE') }}"
-      schema: stripe_integrations_tests_15
+      schema: stripe_integrations_tests_19
       threads: 8
     postgres:
       type: postgres
@@ -42,13 +42,13 @@ integration_tests:
       pass: "{{ env_var('CI_POSTGRES_DBT_PASS') }}"
       dbname: "{{ env_var('CI_POSTGRES_DBT_DBNAME') }}"
       port: 5432
-      schema: stripe_integrations_tests_15
+      schema: stripe_integrations_tests_19
       threads: 8
     databricks:
       catalog: "{{ env_var('CI_DATABRICKS_DBT_CATALOG') }}"
       host: "{{ env_var('CI_DATABRICKS_DBT_HOST') }}"
       http_path: "{{ env_var('CI_DATABRICKS_DBT_HTTP_PATH') }}"
-      schema: stripe_integrations_tests_15
+      schema: stripe_integrations_tests_19
       threads: 8
       token: "{{ env_var('CI_DATABRICKS_DBT_TOKEN') }}"
       type: databricks

docs/index.html

@@ -1,16 +1,16 @@
 config-version: 2
 
 name: 'stripe_integration_tests'
-version: '0.15.1'
+version: '0.16.0'
 
 profile: 'integration_tests'
 
 # For use with validations
 models:
   +schema: "stripe_{{ var('directed_schema','dev') }}"
 
-vars:  
-  stripe_schema: stripe_integrations_tests_15
+vars:
+  stripe_schema: stripe_integrations_tests_19
   stripe__standardized_billing_model_enabled: true
   stripe__using_credit_notes: true
   stripe_source:

docs/manifest.json

@@ -0,0 +1,70 @@
+{{ config(
+    tags="fivetran_validations",
+    enabled=var('fivetran_validation_tests_enabled', false)
+) }}
+
+-- this test ensures the customer_overview end model matches the prior version by comparing the aggregated fields
+-- the below iterates through the prod and dev names to reduce redundancy of logic
+with
+{% for prod_or_dev in ('prod', 'dev') %}
+    {% set cols = [
+        'total_sales',
+        'total_refunds',
+        'total_gross_transaction_amount',
+        'total_fees',
+        'total_net_transaction_amount',
+        'total_sales_count',
+        'total_refund_count',
+        'sales_this_month',
+        'refunds_this_month',
+        'gross_transaction_amount_this_month',
+        'fees_this_month',
+        'net_transaction_amount_this_month',
+        'sales_count_this_month',
+        'refund_count_this_month',
+        'total_failed_charge_count',
+        'total_failed_charge_amount',
+        'failed_charge_count_this_month',
+        'failed_charge_amount_this_month'
+    ] %}
+
+    {{ prod_or_dev }} as (
+        select
+            {% for col in cols %}
+                {% if not loop.first %}, {% endif %}
+                floor(sum({{ col }})) as summed_{{ col }} -- floor and sum is to keep consistency between dev and prod aggs
+            {% endfor %}
+        from {{ target.schema }}_stripe_{{ prod_or_dev }}.stripe__customer_overview
+    ),
+{% endfor %}
+
+prod_not_in_dev as (
+    -- rows from prod not found in dev
+    select * from prod
+    except distinct
+    select * from dev
+),
+
+dev_not_in_prod as (
+    -- rows from dev not found in prod
+    select * from dev
+    except distinct
+    select * from prod
+),
+
+final as (
+    select
+        *,
+        'from prod' as source
+    from prod_not_in_dev
+
+    union all -- union since we only care if rows are produced
+
+    select
+        *,
+        'from dev' as source
+    from dev_not_in_prod
+)
+
+select *
+from final

integration_tests/ci/sample.profiles.yml

@@ -1,2 +1,2 @@
 {% docs source_relation -%} The source where this data was pulled from. If you are making use of the `union_schemas` variable, this will be the source schema. If you are making use of the `union_databases` variable, this will be the source database. If you are not unioning together multiple sources, this will be an empty string.
-{%- enddocs %}
+{%- enddocs %}

integration_tests/dbt_project.yml

@@ -81,21 +81,21 @@ select
     daily_account_balance_transactions.date_day,
     daily_account_balance_transactions.account_id,
     daily_account_balance_transactions.source_relation,
-    coalesce(daily_account_balance_transactions.total_daily_sales_amount/100.0,0) as total_daily_sales_amount,
-    coalesce(daily_account_balance_transactions.total_daily_refunds_amount/100.0,0) as total_daily_refunds_amount,
-    coalesce(daily_account_balance_transactions.total_daily_adjustments_amount/100.0,0) as total_daily_adjustments_amount,
-    coalesce(daily_account_balance_transactions.total_daily_other_transactions_amount/100.0,0) as total_daily_other_transactions_amount,
-    coalesce(daily_account_balance_transactions.total_daily_gross_transaction_amount/100.0,0) as total_daily_gross_transaction_amount,
-    coalesce(daily_account_balance_transactions.total_daily_net_transactions_amount/100.0,0) as total_daily_net_transactions_amount,
-    coalesce(daily_account_balance_transactions.total_daily_payout_fee_amount/100.0,0) as total_daily_payout_fee_amount,
-    coalesce(daily_account_balance_transactions.total_daily_gross_payout_amount/100.0,0) as total_daily_gross_payout_amount,
-    coalesce(daily_account_balance_transactions.daily_net_activity_amount/100.0,0) as daily_net_activity_amount,
-    coalesce((daily_account_balance_transactions.daily_net_activity_amount + daily_account_balance_transactions.total_daily_gross_payout_amount)/100.0, 0) as daily_end_balance_amount,
+    coalesce(daily_account_balance_transactions.total_daily_sales_amount,0) as total_daily_sales_amount,
+    coalesce(daily_account_balance_transactions.total_daily_refunds_amount,0) as total_daily_refunds_amount,
+    coalesce(daily_account_balance_transactions.total_daily_adjustments_amount,0) as total_daily_adjustments_amount,
+    coalesce(daily_account_balance_transactions.total_daily_other_transactions_amount,0) as total_daily_other_transactions_amount,
+    coalesce(daily_account_balance_transactions.total_daily_gross_transaction_amount,0) as total_daily_gross_transaction_amount,
+    coalesce(daily_account_balance_transactions.total_daily_net_transactions_amount,0) as total_daily_net_transactions_amount,
+    coalesce(daily_account_balance_transactions.total_daily_payout_fee_amount,0) as total_daily_payout_fee_amount,
+    coalesce(daily_account_balance_transactions.total_daily_gross_payout_amount,0) as total_daily_gross_payout_amount,
+    coalesce(daily_account_balance_transactions.daily_net_activity_amount,0) as daily_net_activity_amount,
+    coalesce((daily_account_balance_transactions.daily_net_activity_amount + daily_account_balance_transactions.total_daily_gross_payout_amount), 0) as daily_end_balance_amount,
     coalesce(daily_account_balance_transactions.total_daily_sales_count, 0) as total_daily_sales_count,
     coalesce(daily_account_balance_transactions.total_daily_payouts_count, 0) as total_daily_payouts_count,
     coalesce(daily_account_balance_transactions.total_daily_adjustments_count, 0) as total_daily_adjustments_count,
     coalesce(daily_failed_charges.total_daily_failed_charge_count, 0) as total_daily_failed_charge_count,
-    coalesce(daily_failed_charges.total_daily_failed_charge_amount/100, 0) as total_daily_failed_charge_amount
+    coalesce(daily_failed_charges.total_daily_failed_charge_amount, 0) as total_daily_failed_charge_amount
 
 from daily_account_balance_transactions
 left join daily_failed_charges