Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][JsonRowDeserializationSchema] 解析json写入hdfs失败 #460

Open
2 tasks done
Wanglw6 opened this issue Mar 22, 2023 · 0 comments
Open
2 tasks done

[Bug][JsonRowDeserializationSchema] 解析json写入hdfs失败 #460

Wanglw6 opened this issue Mar 22, 2023 · 0 comments
Assignees

Comments

@Wanglw6
Copy link

Wanglw6 commented Mar 22, 2023

What happened

kafka read 配置中"format_type": "json",从Kafka消费json数据写入到hdfs过程中解析json失败。
c2a07f5255d9e4d88bfc2bc961a6fa6
3b52ca52641375a84cbd2be6e06aff8
1679479271640

What do you expect to happen

No response

How to reproduce

{
"job": {
"common": {
"job_id": 333,
"instance_id": 333,
"job_name": "BitSail_kafka_to_hdfs_test",
"user_name": "test"
},
"reader":{
"connector":{
"connector":{
"bootstrap.servers":"PLAINTEXT://hadoop001:6667,hadoop002:6667,hadoop003:6667",
"topic":"test",
"startup-mode":"earliest-offset",
"group":{
"id":"test_consumer_group"
}
}
},
"child_connector_type":"kafka",
"format_type": "json",
"columns": [
{
"name": "ID",
"type": "string"
},
{
"name": "ACCOUNT_ID",
"type": "string"
},
{
"name": "ACCOUNT_PARENT",
"type": "int"
},
{
"name": "ACCOUNT_DESCRIPTION",
"type": "string"
},
{
"name": "ACCOUNT_TYPE",
"type": "string"
},
{
"name": "ACCOUNT_ROLLUP",
"type": "string"
},
{
"name": "CUSTOM_MEMBERS",
"type": "string"
}
],
"class":"com.bytedance.bitsail.connector.legacy.kafka.source.KafkaSourceFunctionDAGBuilder"
},
"writer": {
"hdfs": {
"replication": 3,
"dump_type": "hdfs.dump_type.json",
"compression_codec": "none"
},
"dump": {
"output_dir": "hdfs:///tmp/streaming_file_hdfs/",
"format": {
"type": "hdfs"
}
},
"class": "com.bytedance.bitsail.connector.legacy.streamingfile.sink.FileSystemSinkFunctionDAGBuilder",
"partition_infos": "[{"name":"date","type":"TIME"},{"name":"hour","type":"TIME"}]",
"event_time_fields":"test001",
"rolling.inactivity_interval": 600000,
"rolling.max_part_size": 134217728
}
}
}

Build Environment

No response

Execution Environment

No response

BitSail version

0.1.0

BitSail Component or Code Module

BitSail Core

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants