Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support regex replacement for sink table naming #1241

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

linfan
Copy link

@linfan linfan commented Sep 16, 2022

Problem

Currently this jdbc connect can only sink data to a table with fixed name or use the whole topic name as part of the table name, but in many cases, people may need a more flexible way to named their target tables.

For example, while using debezium source connect, the kafka topic would be named in a format of <server-id>.<source-database>.<source-table>, which is not suitable to be use as part of the sink table name.

Solution

One of the most flexible way for user to specify a table naming rule is regular expression replacement, which means take the kafka topic name and modify it via a regular expression before put it into the final sink table name.

In this PR, an expression of ${topic/Pattern/Replacement/} is supported for advanced sink table naming. This expression format is inspired by sed and bash.

Does this solution apply anywhere else?
  • yes
  • no
If yes, where?

bash support a very similar variable value replacement format as ${var/Pattern/Replacement}, and sed support a more regular expression friendly replacement format as /Pattern/Replacement/, while the char / can be substitute by any other symbol such as #.

Test Strategy

Unit test case is included in this PR.

Testing done:
  • Unit tests
  • Integration tests
  • System tests
  • Manual tests

Release Plan

@linfan linfan requested a review from a team as a code owner September 16, 2022 05:46
@CLAassistant
Copy link

CLAassistant commented Sep 16, 2022

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@OneCricketeer
Copy link

Maybe I'm wrong, but RegexRouter transform is already able to accomplish this?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants