Skip to content

Default partitioning logic is not compatible with Java and kafkajs libraries #121

@themarster

Description

@themarster

After switching to platformatic kafka, we have found that messages are being published to different partitions than they were before.
This breaks compaction and cases of multiple producers with different tech stacks.
The murmur2 implementation seems to be compatible in all three cases, however after applying the murmur2 function, the java and kafkajs libraries use a toPositive implementation as follows:
const toPositive = x => x & 0x7fffffff
(See https://github.com/tulios/kafkajs/blob/master/src/producer/partitioners/legacy/partitioner.js#L12 and https://github.com/apache/kafka/blob/0.10.2/clients/src/main/java/org/apache/kafka/common/utils/Utils.java#L770)
The platformatic library instead uses this:
partition = murmur2(key) >>> 0
(https://github.com/platformatic/kafka/blob/main/src/clients/producer/producer.ts#L337)

If this is intended, it should possibly be added to the readme.

Metadata

Metadata

Assignees

No one assigned

    Labels

    questionFurther information is requested

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions