KafkaJs seems not maintained anymore

[andreacioni]

Is there an existing issue that is already proposing this?

• I have searched the existing issues

Is your feature request related to a problem? Please describe it

Lately I was using NestJs to build a Kafka micro service for a consumer application. I had to study in depth what NestJs uses as the underlying transport to be able to communicate with Kafka brokers safely and efficiently. To be sure of that I needed to tune the underlying Kafka client implementation KafkaJs. While checking the repository I saw that this project is no more maintained and the people who previously worked on it are no longer responding even to discussion they started (see tulios/kafkajs#1603). Because of that I think that this situation could mine the stability of the NestJs project too considering that Kafka is a must-have integration for today's projects.

Describe the solution you'd like

The solution should be to migrate to a stable Kafka implementation that have a strong support. node-rdkafka, which wrap the official C/C++ Confluent driver should be a good choice.

Teachability, documentation, adoption, migration strategy

Dropping the support for KafkaJs may not be the best solution solution for whom is using it without issue in production but certainly you may allow people to start or switch to a different implementation (https://www.npmjs.com/package/node-rdkafka). That could be the right way to start removing support for KafkaJs in case the project remains in this stale situation.

What is the motivation / use case for changing the behavior?

Avoid to be locked in a Kafka implementation that is not going to be supported anymore.

[samuelleach]

Sam from Confluent here.

I know that the Confluent team are now officially supporting a javascript client with this early access release:

https://github.com/confluentinc/confluent-kafka-javascript

From the Readme:

confluent-kafka-javascript is Confluent's JavaScript client for Apache Kafka and the Confluent Platform. This is an early access library. The goal is to provide an highly performant, reliable and easy to use JavaScript client that is based on node-rdkafka yet also API compatible with KafkaJS to provide flexibility to users and streamline migrations from other clients.

Check it out and see if it meets your needs.

[andreacioni]

Hi @samuelleach, thanks for pointing it out. I didn't knew about this work, on Confluent website I just saw a mention to node-rdkafka.

Anyway based on the facts that this library is at an early development stage and the migration from node-rdkafka is going to be trivial (just switching the import statement) I'd suggest for starter to migrate from KafkaJs to node-rdkafka. But That's just my personal opinion.

[trevordixon]

+1 As a heavy user of NestJS and Kafka, I'd like to know the framework uses the best lib. https://github.com/confluentinc/confluent-kafka-javascript says, "This library is currently in early access and not meant for production use." I'd like to see NestJS switch to node-rdkafka sooner than later while keeping an eye on confluent-kafka-javascript for the future.

[andreacioni]

Any thoughts from the maintainers? Because if you decide to change/add node-rdkafka for the underlying implementation of Kafka microservices I'm wondering to propose a PR.

[mkaufmaner]

Sam from Confluent here.

I know that the Confluent team are now officially supporting a javascript client with this early access release:

https://github.com/confluentinc/confluent-kafka-javascript

From the Readme:

confluent-kafka-javascript is Confluent's JavaScript client for Apache Kafka and the Confluent Platform. This is an early access library. The goal is to provide an highly performant, reliable and easy to use JavaScript client that is based on node-rdkafka yet also API compatible with KafkaJS to provide flexibility to users and streamline migrations from other clients.

Check it out and see if it meets your needs.

@samuelleach Looking forward to this library becoming production ready. It is disappointing to see the KafkaJS project become stale and I appreciate Confluent picking it up.

One thing to keep in mind is that NestJS uses a custom partitioner for assigning reply topics to support the MessagePattern. See https://github.com/nestjs/nest/blob/master/packages/microservices/helpers/kafka-reply-partition-assigner.ts

[mkaufmaner]

Related to tulios/kafkajs#1603