Arbitrary message chopping of a TLV format byte stream

[Arenash13]

Good afternoon,

I have a use case where I'm receiving data continuously in a TCP socket.
This data is sent in TLV format (e.g., protobuf, ASN.1, etc.), thus, rather than using a recognisable delimiter between messages (\n, null byte, etc.), the length of a particular PDU is defined in the first X bytes.
I would need a way to retrieve this length s.t. redpanda connect knows where to finish the current message. The fact that the length can vary from one message to another obviously makes the task more complicated 😅.

Do you have any idea on how I could proceed to recover each message with their proper length ?
is it something I can achieve with a custom scanner/processor ?
Are there other implementations that could guide me?

Thanks in advance 😊

[mihaitodor]

Hey @Arenash13 👋 That sounds like a good reason to write a custom scanner. There are examples here on how to write custom plugins. You can probably start by copy / pasting one of the existing scanners, such as chunker, and modifying it according to your needs.

[Arenash13]

Hi @mihaitodor, thanks for the help !