Faust Exploration
Faust is a stream processing library, porting the ideas from Kafka Streams to Python.
It is used at Robinhood to build high performance distributed systems and real-time data pipelines that process billions of events every day.
This repository uses Pipenv for package managment & isolation.
Python version management using asdf is supported via /.tool-versions.
All examples require Kafka, container config is provided in /docker-compose.yaml. Start Kafka:
docker-compose up -d
Run app:
faust -A hello_world worker -l info
Use faust send to send a message to the greetings topic:
faust -A hello_world send greetings "Hello Kafka"
Run app:
faust -A page_views worker -l info
Use kcat to stream messages from page_views-page_views-changelog:
kcat -b localhost -t page_views-page_views-changelog -K :
faust send message to page_views topic:
faust -A page_views send page_views '{"id": "foo", "user": "bar"}'
-
Pypi package
faust-streamingis a fork of the deprecatedfaust -
print()statements are categorized as the "warning" log level, seems weird -
Creates empty folders even when using
store='memory://', not sure why -
In "App Instantiation and Configuration" it says:
Instantiating and setting the configurations of a Faust app can be done in two steps, much like a Flask app. One can create a new App instance with the entire configuration as parameters and then run it, or instantiate the App, set the parameters and then run the application. This allows for easier configuration switching (in particular during development and testing phases).
What does the last sentence mean?