Skip to content

Latest commit

 

History

History
493 lines (296 loc) · 10.2 KB

CHANGELOG.md

File metadata and controls

493 lines (296 loc) · 10.2 KB

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[v2.0.21] 2024-03-25

  • Manually commit uncommitted offsets before closing the Kafka Consumer

[v2.0.18] 2024-03-25

Removed

  • StatsD client
  • kafka.Streams

Changes

  • Internal interface changes for reliablity
  • Improved test cases
  • Smaller API surface area

Breaking changes

  • Streams struct is replaced by kafka.MessageConsumer
  • Streams interface is replaced by MessageConsumer interface
  • StatsD package has been removed
  • Go 1.22 is the min version required
  • ziggurat.RunAll has been removed
  • ziggurat.StartFunc and StopFunc have been removed
  • ziggurat.Event struct has been cleaned up
  • ziggurat.Handler does not return an error anymore

[v1.7.4] 2023-11-01

Changes

  • Upgrades Confluent Kafka client to v2.2.0
  • Fixes RabbitMQ connection URL bug

[1.7.1] 2022-08-05

Changes

  • Update Update github.com/confluentinc/confluent-kafka-go to v1.9.2

[1.7.0] 2022-07-12

Changes

  • Update github.com/confluentinc/confluent-kafka-go to v1.9.1
  • Built-in support for Mac OSX M1 / arm64.

[1.6.5] 2022-05-11

Changes

  • Fixes goroutine publisher loop in statsd

[1.6.3] 2022-05-10

Changes

  • Fixes goroutine leak in statsd publisher

[1.6.2] 2022-05-10

Changes

  • Uses RabbitMQ publisher pooling for publish operations. This significantly improves the publish throughput

[1.6.1] 2022-02-17

Added

  • The function rabbitmq.GetRetryCount is now exposed and can be used to get the retry count without having to do manual assertions

[1.6.0] 2022-02-13

Changes

  • changes internal retry structure for RabbitMQ

Breaking Changes

  • The RabbitMQ QueueConfig now uses QueueKey instead of QueueName

[1.5.1] 2022-01-28

Added

  • consumer config now takes in a new option called PartititionAssignment which is used to configure how kafka partitions are assigned to a consumer

[1.5.0] 2022-01-07

Changes

  • Kafka consumer default PollTimeout is 100 ms
  • ConsumerGroupID is renamed to GroupID and OriginTopics is renamed to Topics to keep it consistent with kafka terminology
  • The run loop for kafka consumer polling has been restructured.

Removed

  • The server package only exposes a Run method which orchestrates an HTTP Server rather than providing a full web server implementation
  • httprouter dependency has been removed

[1.4.5] 2021-12-16

Changes

  • Exposes the ARetry struct from RabbitMQ

[1.4.4] 2021-12-16

Added

  • RabbitMQ MW adds a header retry-origin:ziggurat-go to identify the message retry source
  • Adds a WithHandler option function inside the server package to pass in custom handlers
  • Adds a deprecation comment for ConfigureHandler and ConfigureHTTPEndpoints as this is implementation specific
  • Internal code refactoring
  • Exposes a close method on the statsd client to prevent early closes due to context cancellation
  • StatsD Run method takes in an Option to configure the goroutine publish interval. a negative value will not publish goroutines

[1.4.3] 2021-10-20

Changes

  • fixes go-routine leak in the Dead set replay handler

[1.4.2] 2021-10-19

Breaking changes

  • kafka messages now use the route group to route messages, <route_group>/<topic>/<partition>. Usage of <bootstrap_server>/<consumer_group>/<topic>/<partition> is deprecated as this makes the path very long and routes become difficult to configure.

[1.4.1] 2021-09-09

Breaking changes

  • Compose methods are removed from the router API(s)

Changes

  • uses go v1.17
  • various bug fixes and panic fixes
  • more descriptive errors
  • RabbitMQ's consumers do not use go-routine based workers anymore
  • uses go embed to embed templates to render
  • RabbitMQ WorkerCount is removed from QueueConfig
  • Uses sync.Once to initialize publishers in the Wrap method

Added

  • integration tests for kafka streams and RabbitMQ
  • ziggurat.Use API to compose middleware
  • all packages use the logger.Noop as the default implementation for logging
  • rabbitmq.WithConnectionTimeout to specify a timeout for queue creation
  • Parallelize RabbitMQ consumption using the ConsumerCount

[1.3.5] 2021-08-27

  • adds a rabbitmq dead set replay handler
  • fixes RabbitMQ panic

[1.3.2] 2021-08-25

Added

  • RabbitMQ middleware for auto retrying messages
  • A human friendly logger

Changes

  • Event now includes a metadata map of type map[string]interface{}
  • kafka key values are copied to a new slice before passing it on
  • statsD middleware no longer sends the route label
  • loggers use the logger.DiscardLogger as the default logger

[1.2.2] 2021-07-06

Changes

  • Adds headers to maintain backwards compatibility

[1.2.0] 2021-07-05

Added

  • Regex based kafka router

Changes

  • Moves PipeHandlers to util pkg

[1.1.2] 2021-06-01

Added

  • Makes Prometheus server port configurable

[1.1.1] 2021-05-31

Changes

  • Fixes Prometheus server port

Added

  • An error for streams clean shutdown

[1.0.9] 2021-05-19

Changes

  • fixes parent context propagation in the ziggurat.RunAll method

[1.0.9] 2021-05-14

Changes

  • retract v1.0.8 due to tagging in pipelines

[1.0.8] 2021-05-03

Changes

  • fixes statsD middleware handler metrics publish unit

[1.0.6] 2021-05-02

Added

  • statsD middleware now publishes a generic event delay

[1.0.5] 2021-05-02

Changes

  • statsD middleware only publishes kafka lag and not all events lag
  • fixes handler execution calculation in the statsD middleware

[1.0.4] 2021-04-24

Changes

  • Handler signature now uses the *Event as the second arg
  • Includes a RunAll method to run multiple streams
  • The Run methods returns an error and blocks, leaving concurrency to the caller
  • Kafka polling uses the Poll method instead of the readMessage method

Added

  • statsD middleware
  • Prometheus middleware
  • logging middleware

[1.0.0-alpha.16] 2021-02-23

Added

  • an error for processing failed

[1.0.0-alpha.15] 2021-02-19

Changes

  • streams kafka logs using channels
  • removes rabbitmq middleware
  • removes statsd middleware
  • handler interface is expected to return an error
  • changes the handler signature
  • moves http-server to a new package
  • renames the command from zig to ziggurat

Added

  • interface for structured logging
  • streamer interface
  • event interface

[0.9.0] 2020-12-01

Added

  • consumers process messages concurrently
  • Smaller app API
  • Middleware composition

Changes

  • Refines interface methods
  • fixes tests

[0.8.0] 2020-11-18

Added

  • support for rabbitmq clusters
  • a safer amqp implementation which eliminates the race conditions while re-connecting
  • improves test coverage

Changes

  • reorganised packages
  • provides a message decode hook on MessageEvent

[0.7.1] 2020-11-07

Changes

  • fixes templates

[0.7.0] 2020-11-07

Added

  • updates the kakfa-client lib

Changes

  • uses http.Handler to configure http-routes
  • updates main template
  • defines an interface for App

[0.6.1] 2020-11-06 UNRELEASED

Changes

  • fixes template not found bug
  • uses raw strings instead of tempalte files

[v0.6.0] 20202-11-06

Added

  • adds a sandbox to test out metric publishing,kafka production and retry flow

Changes

  • fixes CLI bugs

[v0.5.2] 2020-11-04

Changes

  • updates ziggurat version in the CLI

[v0.5.1] 2020-11-04

Changes

  • fixes bug in CLI where <- was being escaped

[v0.5.0] 2020-11-04

Added

  • adds new make tasks
  • CLI now generates new compose files for sandbox-ing

Changes

  • renames the MessageRetrier interface to MessageRetry
  • exports handlerFn in the topicEntity struct
  • removes rabbitmq.go
  • uses a thread-safe rabbitmq implementation
  • runs app in async mode

[v0.4.4] 2020-10-28

Changes

  • fixes CLI related issues

[v0.4.0] 2020-10-27

Added

  • Adds a zig CLI to scaffold apps

[v0.3.2] 2020-10-25

Changes

  • fixes httpserver not starting up

[v0.3.1] 2020-10-25

Changes

  • fixes httpserver bug which caused configured routes to be lost

[v0.3.0] 2020-10-25

Added

  • Adds a new middleware to publish message metrics
  • Adds thread safety to MessageAttributes in MessageEvent
  • Adds a metric to measure the handlerFunc exec time
  • Adds a make task to start the app
  • Adds a make task to start-up the metrics containers
  • Adds a make task to produce messages to kafka

Changes

  • halts the app if retries are disabled, and a message is retried
  • changes the --zigurat-config to -config
  • app.Run accepts a zig.RunOptions type
  • fixes race conditions when starting-up the app
  • uses constructors functions to initialize components

[v0.2.0] 2020-10-20

Added

  • Sends app_name as a tag in StatsD

Changes

  • RabbitRetrier uses app context to exit the replay delivery loop
  • App components are not mutable
  • Disables Kafka broker logs

[v0.1.6] - 2020-10-19

Added

  • Add tests for app
  • Add tests for middleware

Changes

  • Passes all middleware args by value
  • Fixes race condition in pipe handlers
  • Makes stop function a part of the app.Run method

[v0.1.5] - 2020-10-16

Added

  • Add tests for util

Changes

  • Fixes middleware execution order
  • Fixes log formatting
  • Disables colored output for logs

[v0.1.4] - 2020-10-15

Added

  • Adds function to DefaultHTTPServer to attach routes
  • Adds ping endpoint to DefaultHTTPServer

[v0.1.3] - 2020-10-14

Changes

  • Retrier interface Start method returns a channel to wait on
  • Rabbit Retrier's consumer polling is moved into Start method

[v0.1.1] - 2020-10-12

Changes

  • Fixes go mod module path

[v0.1.0] - 2020-10-12

Initial release of Ziggurat-golang

Added

  • Kafka consumer group support
  • Middleware support
  • Message retries using RabbitMQ
  • At least once delivery semantics
  • Override config using ENV variables
  • Adds HTTP server for replaying dead set messages
  • Default middleware to deserialize protobuf messages and JSON messages
  • Publish StatsD style counters