You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have an application that reads multiple messages from a Kafka topic. Primarily, these messages are JSON files, which I then parse with the simdjson library. However, for maximum performance the simdjson library requires a string of n bytes to be stored in a buffer of around n + 64 bytes. This means that for any particular message I need to allocate a new buffer for each message and copy over the payload.
Is there a way to make librdkafka allocate extra memory per message when the message is received / taken from the consumer's queue? In particular, I want to have a way to specify that each message (with a payload of n bytes) should allocate n + 64 bytes of storage.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
I have an application that reads multiple messages from a Kafka topic. Primarily, these messages are JSON files, which I then parse with the simdjson library. However, for maximum performance the simdjson library requires a string of
n
bytes to be stored in a buffer of aroundn + 64
bytes. This means that for any particular message I need to allocate a new buffer for each message and copy over the payload.Is there a way to make librdkafka allocate extra memory per message when the message is received / taken from the consumer's queue? In particular, I want to have a way to specify that each message (with a payload of
n
bytes) should allocaten + 64
bytes of storage.Beta Was this translation helpful? Give feedback.
All reactions