Configure producer id/epoch at creation time to reuse idempotency ids from upstream systems

[slinkydeveloper]

Hi everyone,
I have a use case where I have my upstream system that wants to push messages to Kafka in an exactly once fashion. This upstream system has a deduplication mechanism using something like the producer id/epoch (called deduplication id), used to implement exactly once in internal messaging. The state to implement the deduplication is also durably persisted, meaning the system can recover after failures without duplicating messages. Hence my question: would it be possible to bridge our "deduplication ids" to rd_kafka_pid_t used internally in librdkafka idempotent producers to achieve exactly once across system restarts?

I did some investigation, this is what I found:

• rd_kafka_new at some point invokes rd_kafka_idemp_init which down the line sets the producer idempotency state RD_KAFKA_IDEMP_STATE_REQ_PID, requesting a new pid. Once the response is received rd_kafka_idemp_pid_update is invoked and the new pid is set.
• If we introduce a config entry where one can specify producer id and epoch to start from, rd_kafka_new could skip altogether the pid request and simply put itself in the RD_KAFKA_IDEMP_STATE_ASSIGNED state, setting the provided pid.
• Of course when setting this pid manually one needs to be careful about epoch exhaustion and, most important, that the producer id is unique.

Does this makes sense, or is there something I overlooked?

If it can work, would you be interested in supporting such feature? I can provide a patch for this.