Skip to content

Latest commit

 

History

History
75 lines (47 loc) · 1.84 KB

kafkaproducer.md

File metadata and controls

75 lines (47 loc) · 1.84 KB
Raw

Apache-Kafka with dCache, or how to push events outside of dCache.

Starting from version 4.1 we introduced a new approach to external messaging. With the introduction of Apache-Kafka (kafka.apache.org/documentation/) as a message transport, for now in Billing, we start switching to a more modern and efficient messaging system. The goal for the Billing service is easier inclusion with systems like Elasticsearch: Instead of adding messages to a textual log and having an external component parse that log, a Kafka-aware ingester can import them directly into the remote system.

[TOC bullet hierarchy]

What do you need to download to enable Kafka in dCache?

  • ZooKeeper Framework

  • Apache Kafka

Apache Kafka Installation.

1. Download

To install Kafka on your machine, click on the below link − https://www.apache.org/dyn/closer.cgi?path=/kafka/0.9.0.0/kafka_2.11-0.9.0.0.tgz

Extract the tar file

cd opt/
tar -zxf kafka_2.11.0.9.0.0 tar.gz
cd kafka_2.11.0.9.0.0

2. Enable kafka in dCache.

dcache.enable.kafka = true

Set the broker address, or list of broker addresses

dcache.kafka.bootstrap-servers = localhost:9092

Set kafka topic name

dcache.kafka.topic = billing

"billing" is default. The following serice level variables reference dcache.kafka.topic:

dcap.kafka.topic = ${dcache.kafka.topic} ftp.kafka.topic = ${dcache.kafka.topic} nfs.kafka.topic = ${dcache.kafka.topic} pool.kafka.topic = ${dcache.kafka.topic} webdav.kafka.topic = ${dcache.kafka.topic} xrootd.kafka.topic = ${dcache.kafka.topic}

3. Start Server

bin/kafka-server-start.sh config/server.properties

4. Start consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic billing --from-beginning