[flink] Supports debezium-bson format of kafka which collected from mongodb via debezium #4870
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lizc9
changed the title
lizc9
changed the title
lizc9
changed the title
lizc9
force-pushed
the
master
branch
3 times, most recently
from
c37818d to
e96aa44
Compare
yuzelin
reviewed
Jan 9, 2025
...c/main/java/org/apache/paimon/flink/action/cdc/format/debezium/DebeziumBsonRecordParser.java
Outdated
Show resolved
Hide resolved
...c/main/java/org/apache/paimon/flink/action/cdc/format/debezium/DebeziumBsonRecordParser.java
Outdated
Show resolved
Hide resolved
| for (Map.Entry<String, BsonValue> entry : document.entrySet()) { | ||
| String fieldName = entry.getKey(); | ||
| resultMap.put(fieldName, toJsonString(BsonValueConvertor.convert(entry.getValue()))); | ||
| rowTypeBuilder.field(fieldName, DataTypes.STRING()); |
There was a problem hiding this comment.
I think we can map bson type to actual paimon type.
There was a problem hiding this comment.
Because mongodb does not have a schema, the data type of the same field in different documents may be different. For safety reasons, string type is used. By the way, this is the same as mongodb-cdc
There was a problem hiding this comment.
In addition, BsonValueConvertor is used here for data conversion, while mongodb-cdc does not convert data. There will be inconsistencies here, and it may be necessary to discuss.
For example:
The data of mongodb-cdc is:
{
"_id": "{\"$oid\":\"64001c996f4de7ff3189d374\"}",
"updated_at": "{\"$numberLong\":\"1732232838425\"}}"
}The data after BsonValueConvertor conversion is:
{
"_id": "64001c996f4de7ff3189d374",
"updated_at": "1732232838425"
}I think it is also possible to configure whether to use BsonValueConvertor for conversion through TypeMapping
…from mongodb via debezium
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Purpose
Linked issue: open #4615
Tests
API and Format
Documentation