Incorrect key "<null>" in kafka sink when key_field is null

[ruskaof]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Configuring the Kafka sink with key key_field: key results in incorrect key serialization. When an event is sent with key: null, the produced Kafka message contains the string literal "<null>" as the key, rather than a true null value.

This issue happens when a key is set to null explicitly, but when a key is not present in an event, the Kafka sink uses null value as a key as expected.

In the provided configuration, input {"test": "test","key": null} results with a message in Kafka with string literal "<null>" as a key, but input {"test": "test"} results in a message without any key

Configuration

sources:
  http_in:
    type: http_server
    address: 0.0.0.0:5349
    framing:
      method: bytes

transforms:
  decode_message:
    inputs:
      - http_in
    type: remap
    source: |-
      parsed_message = parse_json!(string!(.message))
      .message = parsed_message

sinks:
  kafka_out:
    bootstrap_servers: kafka:9091
    inputs:
      - decode_message
    type: kafka
    topic: test-vector-kafka
    key_field: message.key
    encoding:
      codec: json

Version

0.53.0

Debug Output

Example Data

No response

Additional Context

No response

References

No response

[ruskaof]

Based on the source code of VRL, it looks like the intended behavior (see https://github.com/vectordotdev/vrl/blame/main/src/value/value/serde.rs#L29). However, when working with the Kafka producer, it can lead to unexpected results because the same string literal in the key always results in the same selected partition which I do not expect when I set the key to null