| Kpow Version | Confluent Container Version | Kafka Equivalent |
|---|---|---|
factorhouse/kpow-ce:latest |
confluentinc/cp-kafka:7.7.0 |
org.apache.kafka/kafka:3.7.0 |
- Introduction
- Prequisites
- Run Kafka resources with Kpow Community
- Run Kafka resources with Kpow Enterprise
- Support
- License
This repository contains a Docker Compose environment that will start:
- A 3-node Kafka cluster
- Kafka Connect
- Schema Registry
- Kpow for Apache Kafka, either:
- Kpow Community Edition (free for individuals or organisations), or;
- Kpow Enterprise Trial (supports authentication, RBAC, etc)
All container images used support linux/amd64 and linux/arm64 platforms.
See kafka-local for a simple local configuration consisting of only Kafka.
The local cluster runs with Docker Compose, so you will need to install Docker.
Once Docker is installed, clone this repository and run the following commands from the base path.
git clone [email protected]:factorhouse/kpow-local.git
cd kpow-local
Follow these instructions to run Kafka, Connect, and Schema resources with Kpow Community Edition.
- Get a free Kpow Community license
- Enter the license details into resources/kpow/local-community.env
BOOTSTRAP=kafka-1:19092,kafka-2:19093,kafka-3:19094
CONNECT_REST_URL=http://connect:8083
SCHEMA_REGISTRY_URL=http://schema:8081
SCHEMA_REGISTRY_AUTH=USER_INFO
SCHEMA_REGISTRY_USER=admin
SCHEMA_REGISTRY_PASSWORD=admin
### Your License Details
LICENSE_ID=<license-id>
LICENSE_CODE=<license-code>
LICENSEE=<licensee>
LICENSE_EXPIRY=<license-expiry>
LICENSE_SIGNATURE=<license-signature>
docker compose -f docker-compose-community.yml up
[+] Running 13/13
✔ kpow Pulled 2.7s
✔ schema 11 layers [⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled
...
...
First, hit ctrl-c in the terminal running the Docker Compose process.
^C
Gracefully stopping... (press Ctrl+C again to force)
[+] Stopping 1/3
✔ Container kpow-local-kpow-1 Stopped 0.0s
[+] Stopping 2/3w-local-schema-1 Stopping 0.4s
✔ Container kpow-local-kpow-1 Stopped 0.0s
[+] Stopping 3/3w-local-schema-1 Stopping 0.5s
✔ Container kpow-local-kpow-1 Stopped 0.0s
✔ Container kpow-local-schema-1 Stopped
...Then stop/clear the Docker Compose resources
docker compose -f docker-compose-community.yml down
[+] Running 8/7
✔ Container kpow-local-kpow-1 Removed 0.0s
✔ Container kpow-local-schema-1 Removed 0.0s
✔ Container connect Removed 0.0s
✔ Container kpow-local-kafka-2-1 Removed 0.0s
✔ Container kpow-local-kafka-3-1 Removed 0.0s
✔ Container kpow-local-kafka-1-1 Removed 0.0s
✔ Container kpow-local-zookeeper-1 Removed 0.5s
✔ Network kpow-local_default Removed
The community edition of Kpow for Apache Kafka is free to use by individuals and organisations.
Kpow's UI will be available at http://localhost:3000 once all Kafka resources are running.
To access the cluster you can:
- Connect to the bootstrap on localhost / 127.0.0.1 (most likely non-docker applications)
- Connect to the bootstrap on the Docker defined hosts (kakfa-1, kafka-2, kafka-3)
- Connect to the bootstrap using
host.docker.internalwhich is similar to (1)
Applications that are external to Docker can access the Kafka cluster via the Localhost bootstrap.
bootstrap: 127.0.0.1:9092,127.0.0.1:9093,127.0.0.1:9094
Containerized applications can connect to the Kafka cluster via the Docker Host bootstrap.
These docker hosts (kakfa-1, kafka-2, kafka-3) are defined within the comoose.yml.
When starting your Docker container, specify that it should share the kpow-local_default network.
docker run --network=kpow-local_default ...
Then connect to the hosts that are running on that network
bootstrap: kafka-1:19092,kafka-2:19093,kafka-3:19094
This is a good trick for running a docker container that connects back to a port open on the host machine.
host.docker.internal effective routes back to localhost.
bootstrap: host.docker.internal:9092,host.docker.internal:9093,host.docker.internal:9094
Similarly to Kafka cluster configuration you can access the unauthenticated Kafka Connect cluster via:
http://localhost:8083(plain localhost for non-docker applications)http://connect:8083(when setting the docker network and connecting to the docker host)http://host.docker.internal:8083(when connecting from docker back to the host localhost)
http://localhost:8081(plain localhost for non-docker applications)http://schema:8081(when setting the docker network and connecting to the docker host)http://host.docker.internal:8081(when connecting from docker back to the host localhost)
Each schema client requires the basic authentication credentials of admin/admin.
To add custom Kafka Connect connectors create a resources/connect directory and add all JARs there.
For example, to add the Debezium PostgresSQL connectors:
mkdir -p ./resources/connect
curl -L -o ./resources/connect/debezium-connector-postgres-1.9.6.Final-plugin.tar.gz https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.9.6.Final/debezium-connector-postgres-1.9.6.Final-plugin.tar.gz
cd resources/connect
tar –xvzf debezium-connector-postgres-1.9.6.Final-plugin.tar.gz
Once installed, the connector will be visible within Kpow and available to create.
Follow these instructions to run Kafka, Connect, and Schema resources with Kpow Enterprise Edition.
A trial of Kpow allows you access to all features, including Authentication, Authorisation, RBAC, Audit Log, Prometheus egress, and more.
For example purposes we configure a simple file-based authentication with an example RBAC configuration.
Kpow supports a number of authentication providers including:
- Okta
- OpenID
- OAuth2
- Ldap
- AWS SSO
- Azure AD
- Keycloak
- Get a free Kpow Trial license
- Enter the license details into resoources/kpow/local-trial.env
JAVA_TOOL_OPTIONS=-Djava.security.auth.login.config=/etc/kpow/jaas/hash-jaas.conf
AUTH_PROVIDER_TYPE=jetty
RBAC_CONFIGURATION_FILE=/etc/kpow/rbac/hash-rbac.yml
BOOTSTRAP=kafka-1:19092,kafka-2:19093,kafka-3:19094
CONNECT_REST_URL=http://connect:8083
SCHEMA_REGISTRY_URL=http://schema:8081
SCHEMA_REGISTRY_AUTH=USER_INFO
SCHEMA_REGISTRY_USER=admin
SCHEMA_REGISTRY_PASSWORD=admin
### Your License Details
LICENSE_ID=<license-id>
LICENSE_CODE=<license-code>
LICENSEE=<licensee>
LICENSE_EXPIRY=<license-expiry>
LICENSE_SIGNATURE=<license-signature>
Follow the same start/stop/access method as for Kpow Community with the trial compose configuration, e.g.
docker compose -f docker-compose-trial.yml up
Any issues? Contact support or view our docs.
This repository is released under the Apache 2.0 License.
Copyright © Factor House.