Kafka Connect: Add mechanisms for routing records by topic name #11623
+598
−79
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add 2 new routing mechanisms for records that use Kafka topic name for routing and update configuration for how to route records. The changes move the routing logic to a separate class with different implementations that can be selected via the configuration. It preserves backwards compatibility of current behavior, while adding 2 new ways to route records to tables based on the Kafka topic and also allows custom implementations of how records are routed. The two new routing mechanisms allow the connector to consume from multiple topics, eliminating the need to create separate connectors for each topic to consume from.
|
I wonder if it makes sense to do the routing in the Iceberg kafka connect sink. @bryanck what do you think ? @mun1r0b0t just a note: your PR includes a lot of not necessary change (especially in the md file). Please avoid unnecessary reformating changes as it's harder to review. |
|
Sorry about that. It was my over zealous auto-formatter. Reverted any changes that are not relevant to the PR. |
|
I feel like we can take a much simpler approach for this. For more complex routing needs, an SMT makes more sense to me. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add 2 new routing mechanisms for records that use Kafka topic name for routing and update configuration for how to route records.
The changes move the routing logic to a separate class with different implementations that can be selected via the configuration. It preserves backwards compatibility of current behavior, while adding 2 new ways to route records to tables based on the Kafka topic and also allows custom implementations of how records are routed.
The two new routing mechanisms allow the connector to consume from multiple topics, eliminating the need to create separate connectors for each topic to consume from.
Closes #11163