Group Instance ID support

[deswaan]

What about supporting the Kafka group instance id feature?
This prevents the rebalancing to occur if you keep the same id when you start a new consumer.

[Nevon]

For reference, I believe this is the KIP that is being mentioned: https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances

These are the relevant PRs for the Java client:

1. KAFKA-7862 & KIP-345 part-1: Add static membership logic to JoinGroup protocol apache/kafka#6177
2. KAFKA-8225 & KIP-345 part-2: fencing static member instances with conflicting group.instance.id apache/kafka#6650
3. KAFKA-8220 & KIP-345 part-3: Avoid kicking out members through rebalance timeout apache/kafka#6666
4. KAFKA-8221 & KIP-345 part-4: Add batch leave group request apache/kafka#6714
5. KAFKA-8222 & KIP-345 part 5: admin request to batch remove members based on instance id  apache/kafka#7122

The general gist is to add a new instanceId to the consumer config, which should be included in several APIs, including JoinGroup, LeaveGroup, SyncGroup, OffsetCommit, Heartbeat and DescribeGroup, which means implementing support for those updated API versions. The logic surrounding leaving a consumer group has to also change, as we should not issue a LeaveGroup request if instanceId is set to a non-null value (i.e. this is a static member).

Although I haven't found the Java implementation of it, I also believe that the partition assignment needs to be updated to assign based on the instance id rather than member id, in order to avoid reassigning partitions during rolling restarts.

[tulios]

Just mapping some of the work required, we would need to support protocols JoinGroup v5, SyncGroup v3 (potentially up to v5), OffsetCommit v4, v5, v6, and v7 (potentially v8), Heartbeat v3 and v4, LeaveGroup v2 and v3

[thiagopsnfg]

hello friends, any updates on this issue?

[Nevon]

hello friends, any updates on this issue?

To my knowledge, no one is working on this currently. In #888 I upgraded most of the APIs to their required versions, but nothing has been done beyond that.

[thiagopsnfg]

It's a pity, you did an excellent job @Nevon. Can you measure if it would be too much work for us to try to propose a PR?

[Nevon]

For this kind of change, most of the work is probably in understanding what needs to change. Making the change is usually not too bad. I don't have a full picture of the work, as it's been quite a few months since I last looked at this, but I think it can be done in steps.

1. Implement KIP-320. This is probably gonna be an epic in and of itself. The reason we need this is because we need to support Fetch v12 in order to get the currentLeaderEpoch field, which we need in order to support OffsetFetch v6-7, which in turn we need because it includes the groupInstanceId field that's needed for KIP-345.
2. Add support for OffsetFetch v7
3. Change the behavior of a consumer leaving the group to not issue a LeaveGroup request if the instance id is set.
4. Expose a way of configuring the instance id on the consumer.
5. Update the partition assigner to take into account the instance id (the KIP has a good example of how)

I have not really mapped out the work for KIP-320, so I don't know how big or small it is, but it's definitely the first step needed. We have another issue open for that #818

[jshanman]

Subscribing to this issue. This feature is critical to help us solve rebalancing issues we are seeing during deployments.

[Bessonov]

@jshanman you can find a big button "Subscribe" on the right side. You don't need to post a message and notify all subscribers.

[micheleangioni]

Hey :)
Any news about this topic?

Maybe @tulios or @Nevon could provide an ETA, or just share an update?
We're also affected by rebalancing issues

[Nevon]

As far as I'm aware, no one is actively working on this issue. Neither Tulio nor I are currently working on KafkaJS as part of our jobs, so we can't give any ETAs on anything. I would also like to have this feature, but it doesn't happen unless someone puts the hours in.

[micheleangioni]

I see... thanks for the explanation

[Nevon]

I haven't read through the diff yet, but there is a fork that has implemented some or all of this, referenced in #638 (comment)

[micheleangioni]

It would be amazing if the fork code could be reused in the official library

[sandeepkumarcel]

Opened #1594 for this.

[abbccdda]

Hey there, I'm the original author of static membership. Really appreciate all the efforts here, and I hope @sandeepkumarcel could work with me together to get this landed soon.

[gromas]

Hello. Anything changed in the last 4 years? Thank you.

[augustoViegas2]

hello friends, any updates on this issue?

[roee-hersh]

i cant believe there is no alternative to this library which is widely used but development wise its dead

[RichardWright]

@roee-hersh welcome to nodejs and kafka. The whole thing is a joke and confluent are zero help.

[arrudacaio]

This topic is important to us