Merge pull request #45 from fivetran/release/add-history-mode

[Release] Add daily history models and history functionality to Salesforce transform package

.buildkite/hooks/pre-command

@@ -21,4 +21,5 @@ export CI_SNOWFLAKE_DBT_USER=$(gcloud secrets versions access latest --secret="C
 export CI_SNOWFLAKE_DBT_WAREHOUSE=$(gcloud secrets versions access latest --secret="CI_SNOWFLAKE_DBT_WAREHOUSE" --project="dbt-package-testing-363917")
 export CI_DATABRICKS_DBT_HOST=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HOST" --project="dbt-package-testing-363917")
 export CI_DATABRICKS_DBT_HTTP_PATH=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_HTTP_PATH" --project="dbt-package-testing-363917")
-export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_DBT_TOKEN=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_TOKEN" --project="dbt-package-testing-363917")
+export CI_DATABRICKS_DBT_CATALOG=$(gcloud secrets versions access latest --secret="CI_DATABRICKS_DBT_CATALOG" --project="dbt-package-testing-363917")

.buildkite/pipeline.yml

@@ -58,7 +58,7 @@ steps:
     commands: |
       bash .buildkite/scripts/run_models.sh redshift
 
-  - label: ":bricks: Run Tests - Databricks"
+  - label: ":databricks: Run Tests - Databricks"
     key: "run_dbt_databricks"
     plugins:
       - docker#v3.13.0:
@@ -69,5 +69,6 @@ steps:
             - "CI_DATABRICKS_DBT_HOST"
             - "CI_DATABRICKS_DBT_HTTP_PATH"
             - "CI_DATABRICKS_DBT_TOKEN"
+            - "CI_DATABRICKS_DBT_CATALOG"
     commands: |
       bash .buildkite/scripts/run_models.sh databricks

.buildkite/scripts/run_models.sh

@@ -19,6 +19,8 @@ dbt deps
 dbt seed --target "$db" --full-refresh
 dbt run --target "$db" --full-refresh
 dbt test --target "$db"
-dbt run --vars '{using_account_history_mode_active_records: true, using_opportunity_history_mode_active_records: true, using_user_role_history_mode_active_records: true, using_user_history_mode_active_records: true, salesforce__user_role_enabled: false}' --target "$db" --full-refresh
+dbt run --vars '{salesforce__account_history_enabled: true, salesforce__contact_history_enabled: true, salesforce__opportunity_history_enabled: true}' --target "$db" --full-refresh
+dbt test --target "$db"
+dbt run --vars '{salesforce__account_history_enabled: true, salesforce__contact_history_enabled: true, salesforce__opportunity_history_enabled: true}' --target "$db"
 dbt test --target "$db"
 dbt run-operation fivetran_utils.drop_schemas_automation --target "$db"

CHANGELOG.md

@@ -3,6 +3,44 @@ Please note that this data model will now create a new transformation for **all*
 
 If you are not already a Quickstart Data Model user, you can find out more information [here](https://fivetran.com/docs/transformations/quickstart)!
 
+# dbt_salesforce v1.0.0 
+
+**📣 THIS IS A MAJOR PACKAGE RELEASE! 📣** More details below. 
+
+[PR #45](https://github.com/fivetran/dbt_salesforce/pull/45) includes the following updates:
+
+## 🚨 Breaking Change 🚨
+- We have removed all `tmp` models in the dependent `dbt_salesforce_source` package, and will use the `fivetran_utils.fill_staging_column` macro to compare directly to our source models in your schemas.
+
+## 🚀 Feature Updates 🚀 
+- We have added daily history mode models in the [`models/salesforce_history`](https://github.com/fivetran/dbt_salesforce/tree/main/models/salesforce_history) folder [based off of Fivetran's history mode feature](https://fivetran.com/docs/core-concepts/sync-modes/history-mode), pulling from source models in `dbt_salesforce_source`. This will allow customers to utilize the Fivetran history mode feature, which records every version of each record in the source table from the moment this mode is activated in the equivalent tables.
+
+- **IMPORTANT: All fields in your Salesforce history mode connector that are being synced are being included in the end models**. To change which fields are brought in via end models, you will need to update the fields you are bringing in via your history mode connector in Fivetran and then run a `dbt run --full-refresh`. [See the DECISIONLOG for more details](https://github.com/fivetran/dbt_salesforce_source/blob/main/DECISIONLOG.md).
+
+- Below are the new models included in this update:
+
+|**Model added**|**Description** 
+-----|----- 
+| [salesforce__account_daily_history](https://fivetran.github.io/dbt_salesforce/#!/model/model.salesforce.salesforce__account_daily_history) | Each record is a daily record in an account, starting with its first active date and updating up toward either the current date (if still active) or its last active date.  
+| [salesforce__contact_daily_history](https://fivetran.github.io/dbt_salesforce/#!/model/model.salesforce.salesforce__contact_daily_history) |  Each record is a daily record in an contact, starting with its first active date and updating up toward either the current date (if still active) or its last active date.
+| [salesforce__opportunity_daily_history](https://fivetran.github.io/dbt_salesforce/#!/model/model.salesforce.salesforce__opportunity_daily_history) | Each record is a daily record in an opportunity, starting with its first active date and updating up toward either the current date (if still active) or its last active date. 
+
+- All history models are incremental due to the volume of data being ingested. 
+
+- We support the option to pull from both your standard Salesforce and History Mode connectors simultaneously from their specific database/schemas.  We also support pulling from just your History Mode connector on its own and bypassing the standard connector on its own. [See more detailed instructions for configuring your history mode database and schema variables in the README](https://github.com/fivetran/dbt_salesforce/blob/main/README.md#configuring-your-salesforce-history-mode-database-and-schema-variables).
+
+- These models are disabled by default due to their size, so you will need to set the below variable configurations for each of the individual models you want to utilize in your `dbt_project.yml`. [More details are available in the README](https://github.com/fivetran/dbt_salesforce/blob/main/README.md#enabling-salesforce-history-mode-models).
+
+```yml 
+vars:
+  salesforce__[history_model]_enabled: true ##Ex: salesforce__account_history_enabled: true          
+```
+
+- We've added variable configuration that will allow you to filter the history start and end dates to filter down the data you ingest in each model. See the `Setting the date range for the Salesforce Daily History models` [section in the README](https://github.com/fivetran/dbt_salesforce/blob/main/README.md#filter-your-salesforce-history-mode-models-with-field-variable-conditionals) for more details. 
+
+## 🔎 Under The Hood 🔎
+- We have deprecated the `using_[source]_history_mode_active_records` variables. The introduction of the new history mode capabilities in this package made these variables redundant.  
+
 # dbt_salesforce v0.9.3
 ## 🪲 Bug Fix ⚒️
 [PR #44](https://github.com/fivetran/dbt_salesforce/pull/44) introduces the following update:

DECISIONLOG.md

@@ -0,0 +1,10 @@
+## Syncing all of your fields from the Salesforce History Mode connector
+When creating these new History Mode models, our hypothesis was that the primary reason customers would leverage this data would be to view changes in historical records.
+
+Our normal process is to allow customers to pick and choose the custom fields they bring into their end models. However, omitting any fields that are being synced will lead to new rows of historical records having duplicate data, thus missing out on the potentially. 
+
+Our conclusion was that there is more value for a customer to leverage the history tables if they are syncing all fields they are using, and can thus view all the historical changes in the records they are using. 
+
+There is the drawback of a significant amount of data processing and very large tables with a huge number of columnar values, but we felt this version made the most sense for customers who really want to unlock historical data on their Salesforce tables. 
+
+We are open to feedback on how to improve these history models at all time, [so please contact us directly via our many channels](https://github.com/fivetran/dbt_salesforce_source#-how-is-this-package-maintained-and-can-i-contribute)!