v1.0.0 dbt_fivetran_log

The Fivetran Log connector has been renamed to the "Fivetran Platform" connector. To align with this name change, this package is largely being renamed from fivetran_log to fivetran_platform. This is a very breaking change! 🚨 🚨 🚨 🚨

Bottom Line: What you need to update and/or know:

• If you are setting any variables for this package in your dbt_project.yml, update the name of the prefix of the variable(s) from fivetran_log_* to fivetran_platform_*. The default values for variables have not changed.
• Similarly, any references to package models will need to be updated. The prefix of package models has been updated from fivetran_log__* to fivetran_platform__*.
• If you are overriding the fivetran_log source, you will need to update the overrides property to match the new source name (fivetran_platform).
• Run a full refresh, as we have updated the incremental strategies across warehouses.
• The default build schema suffixes have been changed from _stg_fivetran_log and _fivetran_log to _stg_fivetran_platform and _fivetran_platform respectively. We recommend dropping the old schemas.

Note: Things that are NOT changing in the package:

• The name of the Github repository will not be changed. It will remain dbt_fivetran_log
• The package's project name will remain fivetran_log. You will not need to update your packages.yml reference.
• The default source schema will remain fivetran_log. The name of the source schema variable has changed though (fivetran_log_schema -> fivetran_platform_schema).

See details below!

PR #81 introduced the following changes (some unrelated to the connector name change):

🚨 Breaking Changes 🚨

• Updated the prefixes of each model from fivetran_log_* or stg_fivetran_log_* to fivetran_platform_* and stg_fivetran_platform_*, respectively.

Original model name	New model name
fivetran_log__audit_table	fivetran_platform__audit_table
fivetran_log__connector_daily_events	fivetran_platform__connector_daily_events
fivetran_log__connector_status	fivetran_platform__connector_status
fivetran_log__mar_table_history	fivetran_platform__mar_table_history
fivetran_log__schema_changelog	fivetran_platform__schema_changelog
fivetran_log__transformation_status	fivetran_platform__transformation_status
fivetran_log__usage_mar_destination_history	fivetran_platform__usage_mar_destination_history
stg_fivetran_log__account	stg_fivetran_platform__account
stg_fivetran_log__connector	stg_fivetran_platform__connector
stg_fivetran_log__credits_used	stg_fivetran_platform__credits_used
stg_fivetran_log__destination_membership	stg_fivetran_platform__destination_membership
stg_fivetran_log__destination	stg_fivetran_platform__destination
stg_fivetran_log__incremental_mar	stg_fivetran_platform__incremental_mar
stg_fivetran_log__log	stg_fivetran_platform__log
stg_fivetran_log__transformation	stg_fivetran_platform__transformation
stg_fivetran_log__trigger_table	stg_fivetran_platform__trigger_table
stg_fivetran_log__usage_cost	stg_fivetran_platform__usage_cost
stg_fivetran_log__user	stg_fivetran_platform__user

• Updated the prefix of all package variables from fivetran_log_* to fivetran_platform_*.

Original variable name	New variable name	Default value (consistent)
fivetran_log_schema	fivetran_platform_schema	fivetran_log
fivetran_log_database	fivetran_platform_database	target.database
fivetran_log__usage_pricing	fivetran_platform__usage_pricing	Dynamically checks the source at runtime to set as either true or false. May be overridden using this variable if desired.
fivetran_log__credits_pricing	fivetran_platform__credits_pricing	Dynamically checks the source at runtime to set as either true or false. May be overridden using this variable if desired
fivetran_log_using_sync_alert_messages	fivetran_platform_using_sync_alert_messages	True
fivetran_log_using_transformations	fivetran_platform_using_transformations	True
fivetran_log_using_triggers	fivetran_platform_using_triggers	True
fivetran_log_using_destination_membership	fivetran_platform_using_destination_membership	True
fivetran_log_using_user	fivetran_platform_using_user	True
fivetran_log_[default_table_name]_identifier	fivetran_platform_[default_table_name]_identifier	Default table name (ie 'connector' for fivetran_platform_connector_identifier)

• Updated the default build schema suffixes of package models from _stg_fivetran_log and _fivetran_log to _stg_fivetran_platform and _fivetran_platform respectively.

We recommend dropping the old schemas to eradicate the stale pre-name-change models from your destintation.

• Updated the name of the package's source from fivetran_log to fivetran_platform.
• Updated the name of the packages' schema files:
  • src_fivetran_log.yml -> src_fivetran_platform.yml
  • stg_fivetran_log.yml -> stg_fivetran_platform.yml
  • fivetran_log.yml -> fivetran_platform.yml
• Updated the freshness tests on the fivetran_platform source to be less stringent and more realistic. The following source tables have had their default fresness tests removed, as they will not necessarily update frequently:
  • connector
  • account
  • destination
  • destination_membership
  • user
• Updated the incremental strategy of the audit table model for BigQuery and Databricks users from merge to the more consistent insert_overwrite method. We have also updated the file_format to parquet and added a partition on a new sync_start_day field for Databricks. This field is merely a truncated version of sync_start.
  • Run a full refresh to capture these new changes. We recommend running a full refresh every so often regardless. See README for more details.
• The account_membership source table (and any of its transformations) has been deprecated. Fivetran deprecated this table from the connector in June 2023.

Considerations

• ⚠️ If you are overriding the fivetran_log source, you will need to update the overrides property to match the new source name (fivetran_platform).

Under the Hood

• Added documentation for fields missing yml entries.
• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job (PR #80).
• Updated the pull request templates (PR #80).

Full Changelog: v0.7.4...v1.0.0

v0.7.4 dbt_fivetran_log

PR #79 includes the following updates:

Enhancements

• The sync_id field from the source log table is added to the stg_fivetran_log__log model for ease of grouping events by the sync that they are associated with.

Under the Hood

• Added the get_log_columns macro and included the fill staging cte's within the stg_fivetran_log__log model to ensure the model succeeds regardless of a user not having all the required fields.

Documentation Updates

• The sync_id field is added to the documentation in the fivetran_log.yml file.

Contributors

• @camcyr-at-brzwy (#79)

Full Changelog: v0.7.3...v0.7.4

v0.7.3 dbt_fivetran_log

PR #77 includes the following updates:

Bug Fixes

• The logic within the does_table_exist macro would run the source_relation check across all nodes. This opened dbt compile to erroneous failures in other (non fivetran_log) sources. This macro logic has been updated to only check the source_relation for the specific source in question.
• Adjusted the enabled variable used within the stg_fivetran_log__credits_used model to the more appropriate fivetran_log__credits_pricing name as opposed to fivetran_log__usage_pricing. This ensures a user may override the respective model enablement in isolation of each other.

Documentation Updates

• Added a DECISIONLOG to support the logic behind the fivetran_log__usage_pricing and fivetran_log__credits_pricing variable behaviors within the package.

Full Changelog: v0.7.2...v0.7.3

v0.7.2 dbt_fivetran_log

Bug Fixes

• Fixed duplicated rows in fivetran_log__mar_table_history and set the model back to a monthly granularity for each source, destination, and table. (#74)

Under the Hood

• Adjusted the uniqueness test within the fivetran_log__mar_table_history to also include the schema_name as the same table may exist in multiple schemas within a connector/destination. (#74)

Contributors

• @simon-stepper (#73)

Full Changelog: v0.7.1...v0.7.2

v0.7.1 dbt_fivetran_log

Bug Fixes

• Modified the logic within the fivetran_log__mar_table_history model to no longer filter out previous historical MAR records. Previously, these fields were filtered out as the active_volume source (since deprecated and replaced with incremental_mar) produced a cumulative daily MAR total. However, the incremental_mar source is not cumulative and will need to include all historical records. (#72)

Under the Hood

• Added coalesce statements to the paid_monthly_active_rows and free_monthly_active_rows fields within the fivetran_log__mar_table_history model to coalesce to 0. (#72)

Contributors

• @pkanter (#63)

Full Changelog: v0.7.0...v0.7.1

dbt_fivetran_log v0.7.0

🚨 Breaking Changes 🚨:

PR #68 includes the following breaking changes:

• The active_volume source (and accompanying stg_fivetran_log__active_volume model) has been deprecated from the Fivetran Log connector. In its place, the incremental_mar table (and accompanying stg_fivetran_log__incremental_mar model) has been added. This new source has been swapped within the package to reference the new source table.
  • This new source table has enriched data behind the paid and free MAR across Fivetran connectors within your destinations.
• Removed the monthly_active_rows field from the fivetran_log__mar_table_history and fivetran_log__usage_mar_destination_history models. In it's place the following fields have been added:
  • free_mothly_active_rows: Detailing the total free MAR
  • paid_mothly_active_rows: Detailing the total paid MAR
  • total_mothly_active_rows: Detailing the total free and paid MAR

PR #64 includes the following breaking changes:

• Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically {{ dbt_utils.<macro> }} have been updated to {{ dbt.<macro> }} for the below macros:
  • any_value
  • bool_or
  • cast_bool_to_text
  • concat
  • date_trunc
  • dateadd
  • datediff
  • escape_single_quotes
  • except
  • hash
  • intersect
  • last_day
  • length
  • listagg
  • position
  • replace
  • right
  • safe_cast
  • split_part
  • string_literal
  • type_bigint
  • type_float
  • type_int
  • type_numeric
  • type_string
  • type_timestamp
  • array_append
  • array_concat
  • array_construct
• For current_timestamp and current_timestamp_in_utc macros, the dispatch AND the macro names have been updated to the below, respectively:
  • dbt.current_timestamp_backcompat
  • dbt.current_timestamp_in_utc_backcompat
• dbt_utils.surrogate_key has also been updated to dbt_utils.generate_surrogate_key. Since the method for creating surrogate keys differ, we suggest all users do a full-refresh for the most accurate data. For more information, please refer to dbt-utils release notes for this update.
• packages.yml has been updated to reflect new default fivetran/fivetran_utils version, previously [">=0.3.0", "<0.4.0"] now [">=0.4.0", "<0.5.0"].

dbt_fivetran_log v0.6.4

Happy Wednesday and National Mousse Day 🍫

This release of the fivetran_log package includes the following updates:

Fixes

• Added second qualifying join clause to fivetran_log__usage_mar_destination_history in the usage cte. This join was failing this test to ensure each destination_id has a single measured_month (#67):

      - dbt_utils.unique_combination_of_columns:
          combination_of_columns: 
            - destination_id
            - measured_month 

Under the Hood

• BuildKite testing has been added. (#70)

Contributors

• @lord-skinner (#67)

dbt_fivetran_log v0.6.3

Fixes

• Modified the argument used for the identifier in the get_relation macro used in the does_table_exist macro from name to identifier. This avoids issues on snowflake where the name of a table defined in a source yaml may be in lowercase while in snowflake it is uppercased.

Contributors

• @ggmblr (#60)

dbt_fivetran_log v0.6.2

Happy Thursday 🌻

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• Extend model disablement with config: is_enabled setting in sources to avoid running source freshness when a model is disabled. (#58)

Contributors

• @epapineau (#58)

dbt_fivetran_log v0.6.1

Happy Tuesday! 🌮

This release of the dbt_fivetran_log package includes the following updates:

Fixes

• Added the option to disable the user, account_membership, and destination_membership models that may not be available depending on your Fivetran setup. In your dbt_project.yml you can now change these flags to disable parts of the package. Use the below to configure your project. (#52) and (#55)

  fivetran_log_using_account_membership: false # Disables account membership models
  fivetran_log_using_destination_membership: false # Disables account membership models
  fivetran_log_using_user: false # Disables account membership models

Contributors

• @cmcau (#52)