Skip to content

fede1024/kafka-benchmark

Folders and files

NameName
Last commit message
Last commit date

Latest commit

c04d8ce · Feb 21, 2018

History

25 Commits
Feb 21, 2018
Feb 21, 2018
Nov 19, 2017
Feb 21, 2018
Feb 20, 2018
Sep 14, 2017
Feb 21, 2018

Repository files navigation

kafka-benchmark

Join the chat at https://gitter.im/rust-rdkafka/Lobby

A tool to run programmable benchmarks on Kafka clusters.

This tool uses the high-performance rdkafka Rust client library for Kafka (based on the C library librdkafka), and is able to produce at extremely high speed to the cluster you want to benchmark.

Benchmark scenarios are configurable using a YAML file.

Examples

This is an example run while producing messages to localhost, on an Intel(R) Core(TM) i7-4712HQ CPU @ 2.30GHz:

→ git clone https://github.com/fede1024/kafka-benchmark.git
[...]

→ cd kafka-benchmark/

→ cargo install
[...]

→ kafka-benchmark producer config/base_producer.yaml msg_bursts_base
Scenario: msg_bursts_base, repeat 5 times, 10s pause after each
* Produced 20000000 messages (190.735 MB) in 5.045 seconds using 6 threads
    3964321 messages/s
    37.807 MB/s
* Produced 20000000 messages (190.735 MB) in 5.125 seconds using 6 threads
    3902439 messages/s
    37.217 MB/s
* Produced 20000000 messages (190.735 MB) in 5.032 seconds using 6 threads
    3974563 messages/s
    37.904 MB/s
* Produced 20000000 messages (190.735 MB) in 4.980 seconds using 6 threads
    4016064 messages/s
    38.300 MB/s
* Produced 20000000 messages (190.735 MB) in 5.036 seconds using 6 threads
    3971406 messages/s
    37.874 MB/s
Average: 3964950 messages/s, 37.813 MB/s

→ kafka-benchmark produce config/base_producer.yaml 1KB_bursts
Scenario: 1KB_bursts, repeat 3 times, 20s pause after each
* Produced 200000 messages (1.863 GB) in 2.800 seconds using 6 threads
    71429 messages/s
    681.196 MB/s
* Produced 200000 messages (1.863 GB) in 2.529 seconds using 6 threads
    79083 messages/s
    754.191 MB/s
* Produced 200000 messages (1.863 GB) in 2.514 seconds using 6 threads
    79554 messages/s
    758.691 MB/s
Average: 76492 messages/s, 729.481 MB/s

When producing to localhost, kafka-benchmark can send almost 4 million messages per second on commodity hardware.

About

A tool to run benchmarks on Kafka clusters

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages