Merge pull request #64 from fivetran/MagicBot/dbt-utils-cross-db-migr…

…ation

Updates for dbt-utils to dbt-core cross-db macro migration

CHANGELOG.md

@@ -1,3 +1,44 @@
+# dbt_salesforce_formula_utils v0.8.0
+
+## 🚨 Breaking Changes 🚨:
+[PR #64](https://github.com/fivetran/dbt_salesforce_formula_utils/pull/64) includes the following breaking changes:
+- The default behavior of the `sfdc_formula_view` macro now has the `full_statement_version` equal to `true`. This means, by default the macro will search for the `fivetran_formula_model` source table (full SQL statement for each object) opposed to the `fivetran_formula` source table (individual formulas). 
+  - Please be sure to update your macros that intend to reference the `fivetran_formula` source table accordingly.
+- Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically `{{ dbt_utils.<macro> }}` have been updated to `{{ dbt.<macro> }}` for the below macros:
+    - `any_value`
+    - `bool_or`
+    - `cast_bool_to_text`
+    - `concat`
+    - `date_trunc`
+    - `dateadd`
+    - `datediff`
+    - `escape_single_quotes`
+    - `except`
+    - `hash`
+    - `intersect`
+    - `last_day`
+    - `length`
+    - `listagg`
+    - `position`
+    - `replace`
+    - `right`
+    - `safe_cast`
+    - `split_part`
+    - `string_literal`
+    - `type_bigint`
+    - `type_float`
+    - `type_int`
+    - `type_numeric`
+    - `type_string`
+    - `type_timestamp`
+    - `array_append`
+    - `array_concat`
+    - `array_construct`
+- For `current_timestamp` and `current_timestamp_in_utc` macros, the dispatch AND the macro names have been updated to the below, respectively:
+    - `dbt.current_timestamp_backcompat`
+    - `dbt.current_timestamp_in_utc_backcompat`
+- Dependencies on `fivetran/fivetran_utils` have been upgraded, previously `[">=0.3.0", "<0.4.0"]` now `[">=0.4.0", "<0.5.0"]`.
+
 # dbt_salesforce_formula_utils v0.7.2
 ## Features
 - Incorporated a new boolean argument (`using_quoted_identifiers`) to the `sfdc_formula_view` macro. This argument allows users to quote the compiled sql generated by the macro. This is especially necessary when a users warehouse has case sensitivity enabled. If a user has a warehouse with case sensitivity, they will likely see the macro fail by default. If this is the case, you will want to set the `using_quoted_identifiers` argument to `true` (the argument is `false` by default). ([#72](https://github.com/fivetran/dbt_salesforce_formula_utils/pull/72))

README.md

@@ -1,4 +1,15 @@
-[![Apache License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+<p align="center">
+    <a alt="License"
+        href="https://github.com/fivetran/dbt_salesforce_formula_utils/blob/main/LICENSE">
+        <img src="https://img.shields.io/badge/License-Apache%202.0-blue.svg" /></a>
+    <a alt="dbt-core">
+        <img src="https://img.shields.io/badge/dbt_Core™_version->=1.3.0_<2.0.0-orange.svg" /></a>
+    <a alt="Maintained?">
+        <img src="https://img.shields.io/badge/Maintained%3F-yes-green.svg" /></a>
+    <a alt="PRs">
+        <img src="https://img.shields.io/badge/Contributions-welcome-blueviolet" /></a>
+</p>
+
 # Fivetran Salesforce Formula Utils
 
 This package includes macros to be used within a Salesforce dbt project to accurately map Salesforce Formulas to existing tables.
@@ -12,7 +23,7 @@ This macro is intended to be used within a salesforce dbt project model. To leve
 ```yml
 packages:
   - package: fivetran/salesforce_formula_utils
-    version: [">=0.7.0", "<0.8.0"]
+    version: [">=0.8.0", "<0.9.0"]
 ```
 ## Source Tables Required
 In order to use the macros included in this package, you will need to have a properly configured source package with a source named `salesforce`. An example of a properly configured Salesforce source yml can be found in the `src_salesforce.yml` file in [integration_tests](https://github.com/fivetran/dbt_salesforce_formula_utils/blob/main/integration_tests/models/src_fivetran_formula.yml). This integration_tests folder is just for testing purposes - your source file will need to be in the dbt root models folder. You are welcome to copy/paste the [example](https://github.com/fivetran/dbt_salesforce_formula_utils/blob/main/integration_tests/models/src_fivetran_formula.yml) source configuration into your `src_salesforce.yml` file and modify for your use case. In particular, you will need the following sources defined in your `src_salesforce.yml` file:
@@ -31,9 +42,7 @@ sources:
       ## Any other source tables you are creating models for should be defined here as well. They aren't required it is best organizational practice and  allows Fivetran to compile data lineage graphs
 ```
 ## Model Creation
-Models should be created in the dbt project root models folder. The models should not be named exclusively after the table you are referencing. For example, if you would like to create a model for all formula fields in the account table, if it is named account.sql then this will cause an overwrite on the account table in your destination and require you to resync the data. We recommend naming the model table_name_view.sql as you can see in the [examples](https://github.com/fivetran/dbt_salesforce_formula_utils/tree/main/integration_tests/models) listed in our integration_tests folder.
-
-### (Recommended) Option 1: Generate all relevant formula fields at once
+### (Recommended and default) Option 1: Generate all relevant formula fields at once
 If you would like your model to generate all the formula fields at once related to your source table then you will create a new file in your models folder and name it (`your_table_name_here`.sql). You will then add the below snippet into the file. Finally, update the `source_table` argument to be the source table name for which you are generating the model:
 ```sql
 {{ salesforce_formula_utils.sfdc_formula_view(
@@ -45,15 +54,17 @@ If you would like your model to generate all the formula fields at once related
 If you would like your model to generate all the formula fields related to your source table then you may create a new file in your models folder and name it (`your_table_name_here`.sql). You will then add the below snippet into the file. Finally, update the `source_table` argument to be the source table name for which you are generating the model:
 ```sql
 {{ salesforce_formula_utils.sfdc_formula_view(
-    source_table='your_source_table_name_here') 
+    source_table='your_source_table_name_here',
+    full_statement_version=false) 
 }}
 ```
 ### Option 3: Generate only specified formula fields
 If you would like your model to generate only a specified subset of your formula fields related to your source table then you may create a new file in your models folder and name it (`your_table_name_here`.sql). You will then add the below snippet into the file. Finally, update the `source_table` argument to be the source table name for which you are generating the model and update the `fields_to_include` argument to contain all the fields from your source that you would like to be included in the final output. Be sure that the field(s) you would like to include are enclosed within brackets as an array (ie. `[]`):
 ```sql
 {{ salesforce_formula_utils.sfdc_formula_view(
     source_table='your_source_table_name_here', 
-    fields_to_include=['i_want_this_field','also_this_one','maybe_a_third_as_well','lets_add_more']) 
+    fields_to_include=['i_want_this_field','also_this_one','maybe_a_third_as_well','lets_add_more'],
+    full_statement_version=false) 
 }}
 ```
 ### Formula Fields that Reference Other Formula Fields

dbt_project.yml

@@ -1,5 +1,4 @@
 name: 'salesforce_formula_utils'
-version: '0.7.2'
+version: '0.8.0'
 config-version: 2
-
-require-dbt-version: [">=1.0.0", "<2.0.0"]
+require-dbt-version: [">=1.3.0", "<2.0.0"]

integration_tests/dbt_project.yml

@@ -1,9 +1,7 @@
-
 name: 'salesforce_formula_integration_tests'
-version: '0.7.2'
+version: '0.8.0'
 profile: 'integration_tests'
 config-version: 2
-
 vars:
   salesforce_formula_integration_tests:
     salesforce_schema: salesforce_formula_integrations_tests
@@ -17,4 +15,4 @@ seeds:
         probability: float
     fivetran_formula:
       +column_types:
-        view_sql: "{{ 'varchar(100)'  if target.type in ('redshift','postgres') else 'string'}}"
+        view_sql: "{{ 'varchar(100)'  if target.type in ('redshift','postgres') else 'string'}}"

integration_tests/models/stg_fivetran_formula.yml

@@ -4,9 +4,6 @@ models:
 
   - name: account_view
     description: Represents an individual account, which is an organization or person involved with your business (such as customers, competitors, and partners).
-    tests:
-      - dbt_expectations.expect_table_row_count_to_equal_other_table:
-          compare_model: ref("account")
     columns:
     - name: id
       description: The unique, system-generated ID assigned during creation
@@ -111,9 +108,6 @@ models:
 
   - name: opportunity_view
     description: Represents an opportunity, which is a sale or pending deal.
-    tests:
-      - dbt_expectations.expect_table_row_count_to_equal_other_table:
-          compare_model: ref("opportunity")
     columns:
     - name: id
       description: The unique, system-generated ID assigned during creation.
@@ -192,9 +186,6 @@ models:
 
   - name: user_view
     description: Represents a user in your organization.
-    tests:
-      - dbt_expectations.expect_table_row_count_to_equal_other_table:
-          compare_model: ref("user")
     columns:
     - name: id
       description: The unique, system-generated ID assigned during creation.
@@ -329,9 +320,6 @@ models:
 
   - name: user_role_view
     description: Represents a user role in your organization.
-    tests:
-      - dbt_expectations.expect_table_row_count_to_equal_other_table:
-          compare_model: ref("user_role")
     columns:
     - name: id
       description: The unique, system-generated ID assigned during creation

integration_tests/packages.yml

@@ -1,5 +1,2 @@
 packages:
   - local: ../
-
-  - package: calogica/dbt_expectations
-    version: [">=0.8.0", "<0.9.0"]

integration_tests/requirements.txt

@@ -1,7 +1,7 @@
-dbt-snowflake>=1.0.0,<1.3.0
-dbt-bigquery>=1.0.0,<1.3.0
-dbt-redshift>=1.0.0,<1.3.0
-dbt-postgres>=1.0.0,<1.3.0
-dbt-spark>=1.0.0,<1.3.0
-dbt-spark[PyHive]>=1.0.0,<1.3.0
-dbt-databricks>=1.0.0,<1.3.0
+dbt-snowflake>=1.3.0,<2.0.0
+dbt-bigquery>=1.3.0,<2.0.0
+dbt-redshift>=1.3.0,<2.0.0
+dbt-postgres>=1.3.0,<2.0.0
+dbt-spark>=1.3.0,<2.0.0
+dbt-spark[PyHive]>=1.3.0,<2.0.0
+dbt-databricks>=1.3.0,<2.0.0

macros/sfdc_formula_view.sql

@@ -1,4 +1,4 @@
-{%- macro sfdc_formula_view(source_table, source_name='salesforce', reserved_table_name=source_table, fields_to_include=none, full_statement_version=false, materialization='view', using_quoted_identifiers=false) -%}
+{%- macro sfdc_formula_view(source_table, source_name='salesforce', reserved_table_name=source_table, fields_to_include=none, full_statement_version=true, materialization='view') -%}
 
 -- Best practice for this model is to be materialized as view. That is why we have set that here.
 {{

packages.yml

@@ -1,3 +1,3 @@
 packages:
 - package: fivetran/fivetran_utils
-  version: [">=0.3.0", "<0.4.0"]
+  version: [">=0.4.0", "<0.5.0"]