Consuming from partitions per leader is unbalanced

[xxzefgh]

Looking at implementation here:
https://github.com/platformatic/kafka/blob/main/src/clients/consumer/messages-stream.ts#L364-L406

fetch requests include all assigned partitions for each leader, which results in some partitions advance much faster than the others. Once partitions catch up to current offsets, only then are other partitions allowed to advance. My guess is that if one partition can saturate maxBytes, other partitions get ignored on the server. This differs from Java implementation, where they don't include partitions in a fetch request that already have records in a local buffer.

Quick look at this method should make my point clear:

https://github.com/apache/kafka/blob/b1b330e664ce6380adec873c410a36ef4664206b/clients/src/main/java/org/apache/kafka/clients/consumer/internals/AbstractFetch.java#L421-L488

[ShogunPanda]

I see your point and it makes a lot of sense.
Given Node.js streams work differently, I think we should count messages per topic-partitions in the stream buffer and only advance if they are below a certain threshold (disabled by default to maintain backward compatibility).
I don't think this will be too hard to implement.
Are you willing to send a PR?

[xxzefgh]

@ShogunPanda I think I was wrong on this, after more testing and trying different configurations, this was caused by different Java client defaults, 50MB for maxBytes and 1MB for partitionMaxBytes. I see partitionMaxBytes is not configurable and is set same as maxBytes. I will send a PR for that.

[ShogunPanda]

Amazing, thanks for digging into this!