Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Topic metadata prefetching #1615

Open
mkocsar opened this issue Aug 29, 2023 · 0 comments
Open

Topic metadata prefetching #1615

mkocsar opened this issue Aug 29, 2023 · 0 comments

Comments

@mkocsar
Copy link

mkocsar commented Aug 29, 2023

Is your feature request related to a problem? Please describe.
When a message is being sent to a topic which the KafkaJS Producer hasn't seen before, the topic metadata cache is refreshed which adds to the latency. We are using KafkaJS in a platform where latency is an important concern and producers send messages to thousands of topics, which leads to frequent metadata refreshes and increased latency for those events.

We created a POC where the producer prefetched the metadata for all topics in the cluster before sending any messages and the measured performance improvement was significant:



The script used for measurement can be found at mkocsar/test-prefetch-impact.js (topic setup and teardown scripts at mkocsar/test-send-duration.js). Please note that Improve refreshMetadataIfNecessary in BrokerPool is required to take advantage of metadata prefetching.

Describe the solution you'd like
The way we triggered metadata prefetching in the POC relies on KafkaJS-internal API (cluster.addMultipleTargetTopics) which is normally not accessible by clients. We propose extending the KafkaJS Producer API in a way which allows for prefetching and on-demand refreshing metadata for a given set of topics or all topics of the cluster. The added value would be performance improvements for latency-critical users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant