Add charset conversion feature.

[gnuhpc]

Our senario is to collect to some logs encoded by GBK (a Chinese encoding) rather than UTF-8.

We use flume taildirectorysource plugin to do the collection. After avro serialization, the event was sent to kafka. And then we use logstash-kafka-input to fetch and send it to elasticsearch. The data flow is:

Logs(GBK) -> flume (avro serialization)->kafka-> logstash-kafka-input (avro codec to do the avro deserialization) -> elasticsearch

the input config is:

input {
        kafka {
                bootstrap_servers => "DPFTMP06:9092"
                max_partition_fetch_bytes => "3145728"
                topics =>["emu-topic"]
                group_id => "logstashc"
                auto_offset_reset => "earliest"
                consumer_threads => 8
                key_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
                value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
                charset => "GBK"
                charset_field => ["message"]
                codec => avro {
                        schema_uri => "/logger/logstash/configM/test.avsc"
                }
        }
}

If the logs were UTF-8, it won't cause any trouble. Unfortunately, the logs are GBK. So the kibana will show garbled message when it contains Chinese.

With this pr, we can convert the message to the correct encoding after avro codec in this input plugin.

Please check, thank you!

[cgyim]

@gnuhpc Recently I met that same trouble as you mentioned, but with GB2312 encoding. My data flow is also flume -> kafka -> logstash , flume source: taildir, could you share your configuration of flume ?

[gnuhpc]

@cgyim1992 It's nothing to do with this issue. Please add me in wechat: gnuhpc. Thank you!

[imweijh]

Specify charset GBK in flume config,
Not in logstash-input-kakfa.

[gnuhpc]

@imweijh can you tell me the flume config to configure the GBK charset when using avro? Thank you