v0.11.0 dbt_mixpanel

PR #53 and PR #55 include the following updates:

Feature Update: Run Package on Unioned Connections

• This release supports running the package on multiple Mixpanel sources at once! See the README for details on how to leverage this feature.
  • This was achieved through the introduction of new unioning macros.

Please note: This is a Breaking Change in that we have a added a new field, source_relation, that points to the source connection from which the record originated.
This source_relation field is now part of all generated unique keys.

This will require running a full refresh.

Documentation

• Provided missing column yml documentation.
• Added Quickstart model counts to README. (#56)
• Corrected references to connectors and connections in the README. (#56)

Full Changelog: v0.10.0...v0.11.0

v0.10.0 dbt_mixpanel

PR #49 includes the following updates:

🚨 Breaking Changes 🚨

⚠️ Since the following changes result in the table format changing, we recommend running a --full-refresh after upgrading to this version to avoid possible incremental failures.

• For Databricks All-Purpose clusters, incremental models will now be materialized using the delta table format (previously parquet).

  • Delta tables are generally more performant than parquet and are also more widely available for Databricks users. This will also prevent compilation issues on customers' managed tables.

• For Databricks SQL Warehouses, incremental materialization will not be used due to the incompatibility of the insert_overwrite strategy.

Under the Hood

• The is_incremental_compatible macro has been added and will return true if the target warehouse supports our chosen incremental strategy.
  • This update was applied as there have been other Databricks runtimes discovered (ie. an endpoint and external runtime) which do not support the insert_overwrite incremental strategy used.
• Added integration testing for Databricks SQL Warehouse.
• Added consistency tests for models:
  • mixpanel__daily_events
  • mixpanel__event
  • mixpanel__monthly_events
  • mixpanel__sessions
• Updated logic for macro mixpanel_lookback to align with logic used in similar macros in other packages.

Full Changelog: v0.9.0...v0.10.0

v0.9.0 dbt_mixpanel

PR #41 includes the following updates:

🚨 Breaking Changes 🚨

⚠️ Since the following changes are breaking, a --full-refresh after upgrading will be required.

• Added a default 7-day look-back to incremental models to accommodate late arriving events. The number of days can be changed by setting the var lookback_window in your dbt_project.yml. See the Lookback Window section of the README for more details.

  • Note: This replaces the variable sessionization_trailing_window, which was previously used in the mixpanel__sessions model. This variable was replaced due to the change in the incremental and lookback strategy.

• Performance improvements:

  • Updated the incremental strategy for of the following models to insert_overwrite for BigQuery and Databricks and delete+insert for all other supported warehouses.
    • stg_mixpanel__user_event_date_spine
    • mixpanel__event
    • mixpanel__daily_events
    • mixpanel__monthly_events
    • mixpanel__sessions
  • Removed stg_mixpanel__event_tmp in favor of ephemeral model stg_mixpanel__event. This is to reduce redundancy of models created and reduce the number of full scans.
  • Updated the materialization of stg_mixpanel__user_first_event from a table to a view. This model is used in one downstream model, so a view will reduce storage requirements while not significantly hindering performance.
  • For Snowflake and BigQuery destinations, added cluster_by columns to the configs for incremental models.
  • For Databricks destinations, updated incremental model file formats to parquet for compatibility with the insert_overwrite strategy.

Feature Updates

• Added column dbt_run_date to incremental end models to capture the date a record was added or updated by this package.
• Added _fivetran_id to the mixpanel__event model, since this is the source event table's primary key as of the March 2023 connector release notes.

Contributors

• @jasongroob (#41)
• @CraigWilson-ZOE (#38)

Full Changelog: v0.8.0...v0.9.0

v0.8.0 dbt_mixpanel

This release of dbt_mixpanel includes the following updates:

🎉 Feature Updates 🎉

• Databricks and Postgres compatibility! (PR #33)

Under the Hood:

• Updated incremental strategy for the following incremental models (PR #33):
  • mixpanel__daily_events
  • mixpanel__event
  • mixpanel__monthly_events
  • mixpanel__sessions
  • stg_mixpanel__user_event_date_spine
• Incorporated the new fivetran_utils.drop_schemas_automation macro into the end of each Buildkite integration test job. (PR #32)
• Updated the pull request templates. (PR #32)

Full Changelog: v0.7.0...v0.8.0

dbt_mixpanel v0.7.0

PR #28 includes the following breaking changes:

🚨 Breaking Changes 🚨:

• Dispatch update for dbt-utils to dbt-core cross-db macros migration. Specifically {{ dbt_utils.<macro> }} have been updated to {{ dbt.<macro> }} for the below macros:
  • any_value
  • bool_or
  • cast_bool_to_text
  • concat
  • date_trunc
  • dateadd
  • datediff
  • escape_single_quotes
  • except
  • hash
  • intersect
  • last_day
  • length
  • listagg
  • position
  • replace
  • right
  • safe_cast
  • split_part
  • string_literal
  • type_bigint
  • type_float
  • type_int
  • type_numeric
  • type_string
  • type_timestamp
  • array_append
  • array_concat
  • array_construct
• For current_timestamp and current_timestamp_in_utc macros, the dispatch AND the macro names have been updated to the below, respectively:
  • dbt.current_timestamp_backcompat
  • dbt.current_timestamp_in_utc_backcompat
• dbt_utils.surrogate_key has also been updated to dbt_utils.generate_surrogate_key. Since the method for creating surrogate keys differ, we suggest all users do a full-refresh for the most accurate data. For more information, please refer to dbt-utils release notes for this update.
• Dependencies on fivetran/fivetran_utils have been upgraded, previously [">=0.3.0", "<0.4.0"] now [">=0.4.0", "<0.5.0"].

🎉 Documentation and Feature Updates 🎉:

• Updated README documentation for easier navigation and dbt package setup.
• Included the mixpanel_[source_table_name]_identifier variables for easier flexibility of the package models to refer to differently named sources tables.

dbt_mixpanel v0.6.1

Happy Tuesday 🌮

This release of the dbt_mixpanel package includes the following updates:

Fixes

• Redshift and Postgres warehouses have a limit to the amount of aggregation that may take place within certain functions. The mixpanel__sessions model currently performs a LISTAGG and customers have identified the aggregation sometimes exceeds the limit of the function. Therefore, a conditional was added to check if the target type is Redshift or Postgres. If it is either, it will only perform the aggregation if the count is less than the amount defined by the mixpanel__event_frequency_limit (default 1000) variable. Otherwise, it will return 'Too many event types to render'. (#27)

dbt_mixpanel 0.6.0

🎉 dbt v1.0.0 Compatibility 🎉

🚨 Breaking Changes 🚨

• Adjusts the require-dbt-version to now be within the range [">=1.0.0", "<2.0.0"]. Additionally, the package has been updated for dbt v1.0.0 compatibility. If you are using a dbt version <1.0.0, you will need to upgrade in order to leverage the latest version of the package.
  • For help upgrading your package, I recommend reviewing this GitHub repo's Release Notes on what changes have been implemented since your last upgrade.
  • For help upgrading your dbt project to dbt v1.0.0, I recommend reviewing dbt-labs upgrading to 1.0.0 docs for more details on what changes must be made.
• Upgrades the package dependency to refer to the latest dbt_fivetran_utils. The latest dbt_fivetran_utils package also has a dependency on dbt_utils [">=0.8.0", "<0.9.0"].
  • Please note, if you are installing a version of dbt_utils in your packages.yml that is not in the range above then you will encounter a package dependency error.

Refactor Sessionization Logic

Howdy!

This release addresses two kinds of issues:

1. Some BigQuery users were seeing resources exceeded errors in the mixpanel__sessions model, due to some hefty window functions.
   Fix: date_day is now also incorporated as a partition clause in this model's window functions (example), similar to the de-duping logic. This means that an event that occurs at 11:59pm will NEVER be sessionized with an event that occurs two minutes later at 12:01am.
2. Some folks' event data comes with null device_ids. Device_id was previously relied on as a partition clause in the window functions of mixpanel__sessions, which caused events to never be batched into sessions in this scenario.
   Fix: the sessions model now coalesces device_id and people_id into a user_id column, which is employed as a partition clause in the model's window functions.

dbt 0.20.0 Compatibility

🎉 dbt 0.20.0 Compatibility 🎉

🚨 This is a breaking change! 🚨 dbt v0.20.0 or greater is required for this release. If you are not ready to upgrade, consider using a previous release of this package.

Additional considerations when upgrading to this package:

• This package utilizes the latest v0.7.x release of the dbt-labls/dbt_utils package. If your project also utilizes a version of the fishtown-analytics/dbt_utils package then you will receive a duplicate package error. If this is the case you will need to consider upgrading your other packages to be compatible with this update or use a previous release of this package.
• Similar to the above point, all previous Fivetran dbt packages utilize the fishtown-analytics/dbt_utils package and you will need to upgrade all Fivetran dbt packages to the latest dbt 0.20.0 compatibility releases in order for your packages to run without package conflicts.

Fivetran Utils Dependency Update

🚧 Fivetran Utils Dependency Update 🚧

This release includes the following non-breaking changes:

• Update to the packages.yml to reference the recently released Fivetran Utils package. This is a necessary update to ensure this package is compatible with previous dbt releases before dbt v0.20.0 updates take place.