List of companies using KafkaJS

[Nevon]

Hey KafkaJS users!

We sometimes get questions from people on whether or not KafkaJS is used in production. As the maintainers, we hear from people every now and then, but it's not documented anywhere. Knowing how KafkaJS is used and by whom helps us make decisions and also helps users that feel unsure if they can trust this library or not.

✋ If you or your company is using KafkaJS, please let us know in this thread! ✋

Once we have a couple of users listed, I will turn this into a PR adding the users section to the website.

Known users:

• Klarna - European fintech using KafkaJS across a few dozen services. @tulios & @Nevon

[kpala]

Nightingale Health - Finnish health tech company using KafkaJS in a laboratory information management system. @kpala

[pimpelsang]

Thinking about using in production pipedrive Tallinn, so far only in tests :)

[paambaati]

Indix uses KafkaJS to move massive amounts of data across various systems every day! We also open-sourced the kafkajs-lz4 module that we use in-house.

Indix is a data-as-a-service company that aims to be the Google Maps of product data.

[vykimo]

IBM France Watson
We use it to handle the pipeline of a cognitive platform. 🙂

[arcmode]

Azlo is using KafkaJS to deliver data between micro-services and from micro-services to data lake

[julienvincent]

JourneyApps

We are now using KafkaJS in three areas:

• To power communication and data-replication between our many micro-services.
• To power GraphQL subscriptions for real-time communication between client interfaces
• To power real-time co-editing in our platform IDE

[StevenLangbroek]

We (Quandoo) are writing our first service with it now.

[yarncraft]

As a data engineer at Charp, I am using Kafka.js with Confluent to build an IoT streaming analytical layer.

[StevenLangbroek]

Oh wow totally forgot to update; yeah we've been running this in production for about 4-5 months now :).

Broadly speaking, we consume 3 topics into storage (Postgres & Redis) and allow users to read their data through a GraphQL API.

[ankon]

Good one, also should mention this here:

We (NEXT Technologies) are using KafkaJS to power our platform of about 40 microservices written in TypeScript (event sourcing). We use Kafka hosted in AWS (MSK).

[WaleedAshraf]

Hi everyone,

We at relayr are also planning to switch from Blizzard/node-rdkafka to kafka-node OR KafkaJS. We are looking for something which is production-ready. Blizzard/node-rdkafka has many issues with the consumer getting disconnected and weird error codes and no proper way to disconnect and shutdown the producer/consumer and much more.
We have 20+ Nodejs services that produce and consume messages. We also produce and consume in batch. We are looking for something which also provides some kind of back-pressure or at least the ability to implement it.

If anyone has already done research on pros/cons OR how KafkaJS is a better option, OR if you have faced any issues with kafka-node please share. That would be really helpful.

Thanks,

[kpala]

If anyone has already done research on pros/cons OR how KafkaJS is a better option, OR if you have faced any issues with kafka-node please share. That would be really helpful.

We had kafka-node in use for maybe a year or two since it was the recommended library for Node back then. Lots of connectivity issues and ultimately the only way we could handle these was restarting our services.

Moreover, kafka-node provided multiple options for creating producers and consumers and these options just kept piling up over time without a proper insight which of these we were supposed to be using (we started from HighLevelConsumer and ended up to ConsumerGroupStreams). Now and then I had to go read the source code to understand which options were available since this was not properly documented. This also made me realize the codebase was a mess.

When we were finally fed up with the connectivity issues, we started to migrate to node-rdkafka with equally disappointing results and since we didn't invest much in that migration, we decided to look for yet another library.

With kafkajs our system has been surprisingly stable and we've been happy so far.

However, it's also worth mentioning that during this time we've been improving our infrastructure and Kafka itself has improved a lot as well. Thus, it's not so black-and-white but I wouldn't go back to kafka-node nevertheless.

[WaleedAshraf]

That's really helpful @kpala. I also found some more info, like these benchmarks.
Once I'm done with research and comparison, I'll also share my findings here. 👍

[ThisIsMissEm]

Just to further on @StevenLangbroek's comment, at Quandoo we've also written a bit of a layer on top of kafkajs to help with managing subscriptions to topics, as we've internal standards for our kafka data.

There are still things that we wish were better documented or easier to understand (e.g., a recipes section, like, how do I calculate consumer lag, how do I know when my consumer is "ready" (kappa), and there's a few issues we had around consumer group names getting reused on different topics which leads to only one topic being subscribed)

But overall, kafkajs has been a pleasure to work with & the support from developers outstanding.

[WaleedAshraf]

@ThisIsMissEm for

issues we had around consumer group names getting reused on different topics which leads to only one topic being subscribed

Can you explain a bit more about the issue and how you fixed it?
In our case, we have one consumer-group per service which consumes from different topics. So, we may also run into the same problem if it only consumes from one topic.

[ThisIsMissEm]

@WaleedAshraf it was because we had multiple distinct kafkajs consumers, so called subscribe() multiple times with the same groupId, but different topics.

The solution was to use a unique group per topic as a stop-gap, and then write a consumer manager layer, so we could have different parts of code each connect into the kafka instance, but only one actually subscription.

Writing the I have just realised I could probably instantiate my consumer classes separately, and then pass them into a single "kafka subscriber" method or something (It's a slight inversion of control). I'll try writing it tomorrow maybe.

Basically if you have multiple kafka.consumer's then don't share the groupId; also, you'll notice it not working pretty quickly because you'll see it in the logs.

Which reminds me, I should open-source our small function to reformat kafkajs logs for pino.

[WaleedAshraf]

Thanks for sharing this. 👍👍 I'll test it out tomorrow. Probably we'll have same issue because we use 1 groupId for multiple topics when we subscribe to consumer.

I'm not sure if there's an issue/pr for this to fix within kafkaJS or why there is this limitation.

[ThisIsMissEm]

It's not really a kafkajs issue; it was me misusing the library.

const { Kafka } = require('kafkajs')

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['kafka1:9092', 'kafka2:9092']
})

const consumer1 = kafka.consumer({ groupId: 'test-group' })

await consumer1.connect()
await consumer1.subscribe({ topic: 'test-topic', fromBeginning: true })

const consumer2 = kafka.consumer({ groupId: 'test-group' })

await consumer2.connect()
await consumer2.subscribe({ topic: 'another-topic', fromBeginning: true })

Two kafka.consumers, both sharing the same groupId, but different subscriptions

[tulios]

@WaleedAshraf this is not a KafkaJS limitation, consumer groups can subscribe to multiple topics, but all consumers within the group have to subscribe to the same topics. The concept of a consumer group falls short if every consumer in the group has a different subscription.

If you need help with other questions or need help in general, please, create a specific issue or join out slack channel, let's not flood unrelated issues will all sorts of questions.

Thanks.

[DevNM21-zz]

StudyMonk now uses KafkaJs to track various events and actions performed by the students in our platform

[cwnga2]

We (Oriente) are writing our first service with it now.
this library helps us reduce using callback effort :)
nice~~~

[AustinHunt]

HailTrace

We love, love KafkaJS <3 It is the backbone of our 30+ and growing micro services.

[angelozerr]

@Nevon IMHO I think you should add those projects in your README (we did that for an another project https://github.com/eclipse/lemminx#clients and it gives the capability for a new consumer to know the community which uses your project).

In my side, we use the great kafkajs project inside vscode-kafka and it works like a charm! Thank a lot!

[mateusppereira]

using on production environments here in Quero Education 🇧🇷 🎉

[perniciachd]

I am very new to KafkaJS. I am able to write producers and consumers locally. But, for production deployment, I am not getting any idea of how to deploy either on Azure or AWS. If anyone has suggestions, please share.

[yarncraft]

I am very new to KafkaJS. I am able to write producers and consumers locally. But, for production deployment, I am not getting any idea of how to deploy either on Azure or AWS. If anyone has suggestions, please share.

Here is a list with possible deployment strategies:

Deploy with Cloud Managed Solution [Easy to maintain]:

• https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-for-kafka-ecosystem-overview
• https://www.confluent.io/confluent-cloud/
• https://aiven.io/kafka
• https://aws.amazon.com/msk/

Deploy with Docker [Single Zone]:

• https://www.baeldung.com/ops/kafka-docker-setup
• https://www.kaaiot.com/blog/kafka-docker

Deploy with Kubernetes [Scalable, Multi Zone]:

• https://github.com/banzaicloud/kafka-operator
• https://strimzi.io/
• https://bitnami.com/stack/kafka/helm

I hope this helps!

[AlexanderKaran]

Using it in Production at ClimateClever for communication between our microservices. KafkaJS plus Confluent Cloud integration took me half a day, love it

[LucasKeiti28]

we are using in e-commerce squad at Dasa!

[HeeL]

@ 20 Minuten we were able to achieve great performance and stable results for our analytics system that is now based on KafkaJS ✌️

[feydan]

ONE (banking platform) has been using KafkaJs in our application since inception for asynchronous communication between bounded contexts, concurrent resilient processing of reactions to event log events, and to process external webhook and websocket event data (as well as make it durable).

[Durairaj]

A financial platform where it serves the trading applications, payment applications with over 100+ microservices on different projects. We use Kafkajs for over 2+ years for now.

[Saeger]

We @ Digitalroute - a Swedish company based in Stockholm - has been using KafkaJS with MSK cluster for almost a year now across multiple microservices for our Cloud product, and we're very happy with it :)

[harshagarwal00]

I agree with @ThisIsMissEm please add recipes it would be v useful, for example I found this recipe useful:
#808 (comment)
This would make adoption v quick.. like as a developer..
Im trying to figure...
how to handle long running transactions [this is answered in above link]
how to know if my consumer is lagging.. [still exploring]

[abhishandilya]

GAS is using it for EV charging software

[sandeeprapido]

Rapido is a mobility company in India.

[olasundell]

SVT is the public broadcasting company of Sweden, the Swedish equivalent of the BBC. We use Kafka and kafka.js in our new web publishing system Astrid, a CMS tailor made for our company. Kafka and Postgresql constitutes the backbone of our architecture, and we rely on Kafka's order guarantee to provide simultaneous editing of content items without having to rely on pessimistic locking.

[HeatherFlux]

We use it in prod with confluent cloud at Extend it's used for our event first architecture with confluent. We've actually been building a lot of tooling with it and want to slowly open source some of the IAC tools we built to integrate serverless arch with confluent cloud.

[bkircher]

Knauf is using KafkaJS in their backend systems in all those Node.JS + TypeScript services for async message passing and distributed microservice architecture.