FlowG is a log management platform that lets you ingest, transform, and query logs using a visual pipeline builder. It handles structured logs without requiring predefined schemas and relies on BadgerDB as its storage backend.

For more informations, consult the documentation.

You wish to contribute? Please read our guidelines first.

• Pipelines: Build and manage log processing pipelines visually, with support for transormations with VRL scripts and conditional routing to dedicated streams
• Dynamic Logs: Manage logs with varying structures without needing predefined schemas
• Interoperability: Configuration and log ingestion can entirely be done via the REST API
• Observability: Query and visualize logs in real-time in the integrated Web interface

FlowG's primary goal is to make log refinement and routing as easy as possible. As such, it relies on React Flow to help you build such pipelines with as little code as possible. For the actual code part, we use the Vector Remap Language, which gives a solid base for log refinement.

It aims to replace tools like Logstash by integrating the feature right in the solution.

It also leverages BadgerDB which is a battle tested Key/Value database, with the right feature set to easily support indexing dynamically structured logs, as well as log compression.

Run with (see Build section bellow):

$ task build
$ task bench

System:

• CPU: 13th Gen Intel(R) Core(TM) i7-13700K
• RAM: 64GB

Result:

• Total requests sent: 1,000,000
• Total time: 481.65s
• Average: 2076.18 request/s

Requirements:

• Go 1.22+
• C Toolchain
• Rust and Cargo (edition 2021)
• NodeJS

go install github.com/go-task/task/v3/cmd/task@latest
task build

Then, start the server with:

./bin/flowg serve \
  --auth-dir ./data/auth \
  --log-dir ./data/logs \
  --config-dir ./data/config \
  --http-bind 127.0.0.1:5080 \
  --syslog-bind 127.0.0.1:5514

Now, you can access:

• the WebUI at http://localhost:5080
• the API documentation at http://localhost:5080/api/docs
• the Syslog Server at udp://localhost:5514

A default user root (password: root) and a default pipeline are bootsrapped if no configuration exists during startup.

task docker:build

This will build linksociety/flowg:latest locally.

Then, start the server with:

docker run -p 5080:5080/tcp -p 5514:5514/udp -v flowg-data:/data linksociety/flowg:latest serve

This software is released under the terms of the MIT License