DBT sink + Flink with Kafka consumer reading from Redpanda and sending messages to an http endpoint

[ivanaslamov]

I wanted to create something that potentially can be used as a starting point to implement reverse ETL pipeline using materialized, and redpanda.

In https://github.com/ivanaslamov/mz-hack-day-2022 I introduced a new sink that sends flights data to another redpanda topic (in my project I re-used flight data materializedview as is but it can be any other view). Once back in redpanda the events are then getting consumed by a Flink application that sends them further to an http endpoint.

To run the project first use docker-compose up, then run dbt run like in instructions, list redpanda topics to check name of the materialized view and update it in Application.scala code so it can read from the right place, update http endpoint in the flink http sink code and restart flink container.

Note for future, need to better familiarize myself with materialize TAIL mechanism to potentially improve that implementation further.

[morsapaes]

Thanks for sharing your project, @ivanaslamov! Using TAIL should get you there without the need to spin up a Flink application on the side (it's cool that you found a way to follow through, though).

Let us know if you give it another go! @bobbyiliev and @joacoc are more proficient with TAIL than I'll ever be, but we're all around in the Materialize Community Slack!