librdkafka - the Apache Kafka C/C++ client library

Copyright (c) 2012-2018, Magnus Edenhill.

https://github.com/edenhill/librdkafka

librdkafka is a C library implementation of the Apache Kafka protocol, containing both Producer and Consumer support. It was designed with message delivery reliability and high performance in mind, current figures exceed 1 million msgs/second for the producer and 3 million msgs/second for the consumer.

librdkafka is licensed under the 2-clause BSD license.

For an introduction to the performance and usage of librdkafka, see INTRODUCTION.md

See the wiki for a FAQ.

NOTE: The master branch is actively developed, use latest release for production use.

Overview

• High-level producer
• High-level balanced KafkaConsumer (requires broker >= 0.9)
• Simple (legacy) consumer
• Compression: snappy, gzip, lz4, zstd
• SSL support
• SASL (GSSAPI/Kerberos/SSPI, PLAIN, SCRAM) support
• Broker version support: >=0.8 (see Broker version compatibility)
• Stable C & C++ APIs (ABI safety guaranteed for C)
• Statistics metrics
• Debian package: librdkafka1 and librdkafka-dev in Debian and Ubuntu
• RPM package: librdkafka and librdkafka-devel
• Gentoo package: dev-libs/librdkafka
• Portable: runs on Linux, OSX, Win32, Solaris, FreeBSD, AIX, ...

Language bindings

• C#/.NET: confluent-kafka-dotnet (based on rdkafka-dotnet)
• C++: cppkafka
• D (C-like): librdkafka
• D (C++-like): librdkafkad
• Erlang: erlkaf
• Go: confluent-kafka-go
• Haskell (kafka, conduit, avro, schema registry): hw-kafka
• Lua: luardkafka
• Node.js: node-rdkafka
• OCaml: ocaml-kafka
• PHP: phpkafka
• PHP: php-rdkafka
• Python: confluent-kafka-python
• Python: PyKafka
• Ruby: Hermann
• Ruby: rdkafka-ruby
• Rust: rust-rdkafka
• Tcl: KafkaTcl
• Swift: Perfect-Kafka

Users of librdkafka

• kafkacat - Apache Kafka swiss army knife
• Wikimedia's varnishkafka - Varnish cache web log producer
• Wikimedia's kafkatee - Kafka multi consumer with filtering and fanout
• rsyslog
• syslog-ng
• collectd
• logkafka - Collect logs and send to Kafka
• redBorder
• Headweb
• Produban's log2kafka - Web log producer
• fuse_kafka - FUSE file system layer
• node-kafkacat
• OVH - AntiDDOS
• otto.de's trackdrd - Varnish log reader
• Microwish has a range of Kafka utilites for log aggregation, HDFS integration, etc.
• aidp - kafka consumer embedded Lua scripting language in data process framework
• Yandex ClickHouse
• NXLog - Enterprise logging system, Kafka input/output plugin.
• large unnamed financial institutions
• and many more..
• Let me know if you are using librdkafka

Usage

Requirements

The GNU toolchain
GNU make
pthreads
zlib-dev (optional, for gzip compression support)
libssl-dev (optional, for SSL and SASL SCRAM support)
libsasl2-dev (optional, for SASL GSSAPI support)
libzstd-dev (optional, for ZStd compression support)

NOTE: Static linking of ZStd (requires zstd >= 1.2.1) in the producer enables encoding the original size in the compression frame header, which will speed up the consumer. Use STATIC_LIB_zstd=/path/to/libzstd.a ./configure --enable-static to enable static ZStd linking. MacOSX example: STATIC_LIB_zstd=$(brew ls -v zstd | grep libzstd.a$) ./configure --enable-static

Instructions

Installing prebuilt packages

On Mac OSX, install librdkafka with homebrew:

$ brew install librdkafka

On Debian and Ubuntu, install librdkafka from the Confluent APT repositories, see instructions here and then install librdkafka:

$ apt install librdkafka-dev

On RedHat, CentOS, Fedora, install librdkafka from the Confluent YUM repositories, instructions here and then install librdkafka:

$ yum install librdkafka-devel

On Windows, reference librdkafka.redist NuGet package in your Visual Studio project.

For other platforms, follow the source building instructions below.

Building

  ./configure
  # Or, to automatically install dependencies using the system's package manager:
  # ./configure --install-deps
  # Or, build dependencies from source:
  # ./configure --install-deps --source-deps-only

  make
  sudo make install

NOTE: See README.win32 for instructions how to build on Windows with Microsoft Visual Studio.

NOTE: See CMake instructions for experimental CMake build (unsupported).

Usage in code

See examples/rdkafka_example.c for an example producer and consumer.

Link your program with -lrdkafka -lz -lpthread -lrt.

Documentation

The public APIs are documented in their respective header files:

• The C API is documented in src/rdkafka.h
• The C++ API is documented in src-cpp/rdkafkacpp.h

To generate Doxygen documents for the API, type:

make docs

Configuration properties are documented in CONFIGURATION.md

For a librdkafka introduction, see INTRODUCTION.md

Examples

See the examples/sub-directory.

Tests

See the tests/sub-directory.

Support

File bug reports, feature requests and questions using GitHub Issues

Questions and discussions are also welcome on irc.freenode.org, #apache-kafka, nickname Snaps.

Commercial support

Commercial support is available from Edenhill services