The repo is versioned based on SemVer 2.0 using the tiny-but-mighty MinVer from @adamralph. See here for more information on how it works.
All notable changes to this project will be documented in this file. The format is based on Keep a Changelog.
The Unreleased
section name is replaced by the expected version of next release. A stable version's log contains all changes between that version and the previous stable version (can duplicate the prereleases logs).
Feed
:Monitor.AwaitCompletion
enables quasi deterministic waiting for the processing of async reactions within integration tests #170Feed
:Checkpoint
enables committing progress (and obtaining the achieved positions) without stopping the Sink #162Feed.SinglePassFeedSource
: Coordinates reads of a set of tranches until each reaches its Tail #179Scheduler
: Split out stats rerateLimited
andtimedOut
vsexceptions
#194Scheduler
:purgeInterval
to control memory usage #97Scheduler
:wakeForResults
option to maximize throughput (without having to drop sleep interval to zero) #161Sinks
,Sinks.Config
: top level APIs for wiring common sink structures #208Ingester
: Expose optionalingesterStatsInterval
control #154Internal.Async.AwaitKeyboardInterruptAsTaskCanceledException
: Honored cancellation; moved toPropulsion.Internal
#160Propulsion.EventStoreDb
: PortedEventStore
to targetEquinox.EventStore
>=4.0.0
(using the gRPC interface) #139Propulsion.CosmosStore3
: Special cased version ofPropulsion.CosmosStore
to targetEquinox.CosmosStore
v[3.0.7
-3.99.0]
Deprecated; Please migrate toPropulsion.CosmosStore
by updatingEquinox.CosmosStore
dependencies to4.0.0
#139Propulsion.DynamoStore
:Equinox.CosmosStore
-equivalent functionality forEquinox.DynamoStore
. Combines elements ofCosmosStore
,SqlStreamStore
,Feed
#140 #140 #177Propulsion.MessageDb
:FeedSource
andCheckpointStore
for MessageDb #181 🙏 @nordfjordPropulsion.MemoryStore
:MemoryStoreSource
to align with other sources for integration testing. Includes deterministicAwaitCompletion
as perPropulsion.Feed
-based Sources #165Propulsion.SqlStreamStore
: AddedstartFromTail
#173Propulsion.Tool
:checkpoint
commandline option; enables viewing or overriding checkpoints #141Propulsion.Tool
: Add support for autoscaling throughput of Cosmos containers and databases #142 🙏 @brihadish
- Targeted
net6.0
with6.0.300
SDK,FSharp.Core
v6.0.0
,FSharp.Control.TaskSeq
v0.4.0-alpha.1
,MathNet.Numerics
v4.15.0
- Changed all
Tuple
types tostruct
tuples (System.ValueTuple
) #169 - Changed all
Async
toTask
#192 Dispatcher
: ReplacedGetConsumingEnumerable
withSystem.Threading.Channels
#155Ingester
,Submitter
: ReplacedAsync.Sleep
withTask.WhenAny
; Condensed logging #154Pipeline
: RenamedAwaitShutdown
toAwait
, addedWait
#192Projector
: ReplacedAsync
withtask
for projector supervision #156Scheduler
: ReplacedConcurrent*
withSystem.Threading.Channels
#156Scheduler
:Struct
/voption
conversions; buffer reuse #157Scheduler
: ReplacedThead.Sleep
withTask.WhenAny
; Added Sleep time logging #161Streams
: Changed dominantITimelineEvent
EventBody
type frombyte[]
toSystem.ReadOnlyMemory<byte>
(Sinks.EventBody
) #169 #208Streams.SpanResult
: Renamed toSinks.StreamResult
#208Propulsion.CosmosStore
: Changed to targetEquinox.CosmosStore
v4.0.0
#139Propulsion.CosmosStore.CosmosSource
: Changed parsing to useSystem.Text.Json
#139Propulsion.EventStore
: Pinned to targetEquinox.EventStore
v[3.0.7
-3.99.0]
Deprecated; Please migrate toPropulsion.EventStoreDb
#139Propulsion.EventStoreDb.EventStoreSource
: Changed API to matchPropulsion.SqlStreamStore
API rather thanPropulsion.EventStore
#139Propulsion.Feed
,Kafka
: ReplacedAsync
withtask
for supervision #158, #159Propulsion.Kafka
: TargetFsCodec.NewtonsoftJson
v3.0.0
#139
Streams.StreamSpan
: Changed from a record to individual arguments ofFsCodec.StreamName
andSinks.Event[]
#169 #208Streams
:statsInterval
is obtained from theStats
wherever one is supplied #208Propulsion.Cosmos
: Should not be in general use - users should port toPropulsion.CosmosStore3
, thenPropulsion.CosmosStore
#193Destructurama.FSharp
dependency #152Ingester
,Submitter
: Removed requirement to specify sleep intervals #154Scheduler
: Removed need to tuneidleDelay
except in extreme circumstances #161
Propulsion.SqlStreamStore
: Replaced incorrect/meaningless stream name forSqlStreamStoreSource
#139- Synced
AwaitTaskCorrect
with official version 3c11142 Projector
,Ingester
,Submitter
,Scheduler
: Deterministic shutdown via Cancellation and/or unhandled exceptions #154
2.12.2 - 2022-03-10
Kafka
: TargetFsCodec.NewtonsoftJson
v2.3.2
#138
2.12.1 - 2022-02-15
Prometheus
:Propulsion.Streams.Log.LogSink
: Support indicating the consumer group name via a Serilog (ForContext
) property"group"
#137
Prometheus
:Propulsion.Streams.Log.LogSink
: Replace mandatorygroup
argument with optionaldefaultGroup
to emphasize primacy of"group"
Log property #137
2.12.0 - 2022-01-28
Kafka.Producers
: optional constructor parameters #135 🙏 avsaditya19
EventStore
: TargetEquinox.EventStore
v3.0.6
#136
2.12.0-rc.3 - 2021-12-24
EventStore
: TargetEquinox.EventStore
v3.0.5
#134
2.12.0-rc.2 - 2021-12-20
CosmosStoreSource
: GeneralizeRun
with integrated Ctrl-C handling toStart
yielding aPipeline
#133
2.12.0-rc.1 - 2021-12-06
Streams
: Addedpropulsion_scheduler_busy
metrics:count
andseconds
#126
Feed.PeriodicSource
: AddreadTranches
parameter toPump
#130Feed.PeriodicSource
: AddTrancheId
parameter tocrawl
#130CosmosStore
: TargetEquinox.CosmosStore
v3.0.5
SqlStreamStore
: Rebase onPropulsion.Feed.FeedSource
#131
net461
support #123
StreamsProjector
: Fix failure to pump batches when usingmaxReadAhead
ofInt32.MaxValue
#124
2.11.0 - 2021-10-19
CosmosStore
: TargetMicrosoft.Azure.Cosmos
(V3 CFP)3.0.22
2.11.0-rc3 - 2021-10-18
Feed.PeriodicSource
: Enables periodic traversal of a source where the source does not present an incrementally readable change feed #117Feed.Prometheus
: Prometheus integration providing metrics not dissimilar to those available from.CosmosStore
(no lag metrics though) #117Pipeline
: AddedAwaitWithStopOnCancellation
#118CosmosStore
: Added handling ofWith(Error|LeaseAcquire|LeaseRevoke)Notification
APIs introduced inMicrosoft.Azure.Cosmos
v3.22.0-preview
#120
CosmosStore
: TargetMicrosoft.Azure.Cosmos
(V3 CFP)3.0.22-preview
CosmosStore
: TargetEquinox.CosmosStore
v3.0.5
CosmosStore
: renamemaxDocuments
tomaxItems
Pipeline
: RenamedAwaitCompletion
toAwaitShutdown
#118Kafka
: TargetFsKafka [1.7.0, 1.9.99)
Kafka0
: Triggered by removal of underlyingFsKafka0
#119Kafka
: Removed support fornet461
, triggered by removal of same in underlyingFsKafka
#119
- Fix CI to stop emitting builds with incorrect
AssemblyVersion 1.0.0.0
(updated MinVer to2.5.0
) - Update global.json to use SDK version
5.0.200
2.11.0-rc2 - 2021-06-22
CosmosStore
: TargetMicrosoft.Azure.Cosmos
(V3 CFP)3.0.20-preview
#115
2.11.0-rc1 - 2021-06-21
CosmosStore
: Clean/reorg in preparation for porting to V3 SDK; targetEquinox.CosmosStore
v3.0.1
#114CosmosStore
: Port fromMicrosoft.Azure.Documents.ChangeFeedProcessor
(V2 CFP) toMicrosoft.Azure.Cosmos
(V3 CFP)3.0.19-preview1
#113
2.10.0 - 2021-05-13
CosmosStore
: TargetEquinox.CosmosStore
v3.0.0
Cosmos
: TargetEquinox.Cosmos
v[2.6.0,2.99.0]
EventStore
: TargetEquinox.EventStore
v[2.6.0,2.99.0]
2.10.0-rc10 - 2021-05-05
Cosmos
: Replace incomplete/inconsistent usage ofpartitionKeyRangeId
and"Range"
withpartitionId
and"Reader"
in ChangeFeed ingestion path #112Ingester
: Replace"Uncommitted" with "Reader" and "Ahead"
terminology, and includepartitionId
in all messages #112
Kafka.StreamsConsumerStats
: replaced byPropulsion.Streams.Stats
#111
2.10.0-rc9 - 2021-04-30
*.Prometheus.LogSink
: Generalizedapp
tag to arbitrary custom tags as per Equinox #287 #109 🙏 @deviousasti
2.10.0-rc8 - 2021-04-06
Propulsion.Feed
: Initial implementation of generic feed consumer with support for checkpointing in an Equinox stream (HT @scrwtp who provided the baseline impl inPropulsion.SqlStreamStore
) #106
- Moved
Propulsion.Streams.Projector.Stats
toPropulsion.Streams.Stats
#108
2.10.0-rc7 - 2021-03-15
StreamsProjector
: Tuned submission policy to forward 80% of batches read ahead (except in Kafka case where there's limited benefit so default remains to submit 5 per partition) #105
2.10.0-rc6 - 2021-03-13
Propulsion.CosmosStore
: Included select aspects ofEquinox.Cosmos
constructs wrt connection logic etc; required until theCosmosStoreSource
impl uses V >=3CosmosClient
#104
2.10.0-rc5 - 2021-03-12
Propulsion.CosmosStore.CosmosStoreSource
- equivalent of feature inPropulsion.Cosmos
#103
2.10.0-rc4 - 2021-03-12
Propulsion.CosmosStore.CosmosStoreSink
+CosmosStorePruner
- equivalents of features inPropulsion.Cosmos
#89StreamSpan.Version
,SpanResult.NoneProcessed
#102
2.10.0-rc3 - 2021-03-04
- Streams Scheduler: Tune memory consumption re write positions of inactive streams #94
2.10.0-rc2 - 2021-01-27
Propulsion.Prometheus
: Add Buffer, Cpu and Handler latency metrics toPropulsion.Streams
Scheduler #93
- Add
<RollForward>Major</RollForward>
forPropulsion.Tool
#92
- Replaced numeric field names with strings in latency percentiles message #93
2.10.0-rc1 - 2020-12-03
Propulsion.Cosmos
:prometheus-net
integration #91
Kafka
: Replace defaulting ofProducer
'sacks
defaulting toLeader
with mandatory argument #89
2.9.1 - 2020-11-09
Cosmos.CosmosPruner
: Fixed off-by-one error that renders pruning inert #88
2.9.0 - 2020-11-04
- Exposed optional
maxEvents
andmaxBytes
arguments forCosmosSink
#84
Cosmos
: Tweak logging inCosmosSink
#85
2.9.0-rc2 - 2020-09-08
- Consistently exposed optional
idleDelay
andpumpInterval
arguments for all Projectors and Ingesters #81
Kafka.StreamsProducerSink
: Changed bad defaultidleDelay
from 0ms to 1ms #82
2.9.0-rc1 - 2020-08-31
Cosmos
: AddCosmosPruner
#80
- Corrected Cosmos + Stream Sink error logging #79
2.8.1 - 2020-08-04
Kafka.Producer
: Exposeacks
,linger
,compression
options #78
Kafka
: Adjust default linger to 5ms #78
- Apply Corrected FsKafka Poll loop when over
maxInFlightBytes
threshold as perFsKafka
v1.5.0
#77
2.8.0 - 2020-07-27
Kafka
: TargetsFsKafka
/FsKafka0
(andConfluent.Kafka
) v1.5.0
#75
2.7.1 - 2020-07-27
2.7.0 - 2020-06-09
2.6.0 - 2020-05-20
Kafka
: AddPropulsion.Kafka.ConsumeResultContext
and defaultStreamNameSequenceGenerator.ConsumeResultToStreamEvent
#70
Kafka
: TargetsFsKafka
/FsKafka0
(andConfluent.Kafka
) v1.4.3
#69
2.5.2 - 2020-05-16
Kafka
: TargetsFsKafka
/FsKafka0
v1.5.0-beta.2
to resolveTypeLoadException
when using1.5.0-beta.1
2.5.1 - 2020-05-14 (Unlisted/broken : ☝️)
EventStoreSource
: Fixedobj
->int
type discrepancy re #63
2.5.0 - 2020-05-13 (Unlisted/broken ☝️)
Kafka
: TargetsFsKafka
/FsKafka0
v1.4.2
#67
Propulsion.Kafka0
SomePropulsion.Kafka0
-namespaced shimming elements are now found in theFsKafka
namespace inFsKafka0
#67Propulsion.Kafka
:KafkaMonitor
is now found in theFsKafka
namespace inFsKafka
/FsKafka0` (NOTE: integration tests continue to live in this repo) #67
Kafka
: Change buffer grouping to includeTopic
alongsidePartitionId
- existing implementation did not guarantee marking progress where consuming from more than one Topic concurrently #63
2.4.3 - 2020-04-27
- Revised
handle
function signatures inPropulsion.Sync.StreamsSync
andPropulsion.Streams.StreamsProjector
to usePropulsion.Streams.SpanResult
representing Write Position updates #62
2.4.2 - 2020-04-27
- Mark
Scheduler.Stats
abstract, reorder/namestatsInterval
arguments #61
2.4.1 - 2020-04-25
- Added abstract
HandleExn
method totype Stats
; removed defaulting ofstats
arguments in almost all cases #60
2.4.0 - 2020-04-24
- BREAKING: Revised
handle
function signatures inPropulsion.Kafka.StreamsConsumer
andPropulsion.Streams.StreamsProjector
to include aPropulsion.Streams.SpanResult
representing Write Position updates #59
- Removed egregious
int64
from stats handler signatures inPropulsion.Streams.Scheduling.StreamSchedulerStats
andProjector.Stats
#59 - Renamed
Streams.Sync.StreamsSyncStats
andStreams.Scheduling.StreamSchedulerStats
toStats
for consistency #59
2.3.0 - 2020-04-22
Propulsion.Kafka
:StreamNameSequenceGenerator.ConsumeResultToStreamEvent
provides a set of helpers for mapping from KafkaConsumeResult
s to aStreamEvent
for processing #56Propulsion.Kafka.StreamsConsumer.Start
: now has aconsumeResultToStreamEvents : ConsumeResult<_, _> -> Propulsion.Streams.StreamEvent<_> seq
parameter (which can be implemented viaStreamNameSequenceGenerator.ConsumeResultToStreamEvent
) #56- BREAKING: Moved pubternal
Propulsion.Kafka.Core.StreamKeyEventSequencer
to publicPropulsion.Kafka.StreamNameSequenceGenerator
#56
- BREAKING: Moved
Propulsion.Kafka.StreamsConsumer
's Start methods toPropulsion.Core.StreamsConsumer
(i.e. need to prefixStreamsConsumer
to migrate) #56 - BREAKING:
Propulsion.Kafka.Core.StreamKeyEventSequencer.ToStreamEvent
becomesPropulsion.Kafka.StreamNameSequenceGenerator.KeyValueToStreamEvent
#56
2.2.0 - 2020-04-18
Propulsion.Kafka
/Kafka0
: ExposedProduce
API providing ability to customize theConfluent.Kafka
API usedPropulsion.Kafka
: AddedProduce
API admitting a rawMessage
Propulsion.Kafka
: AddedProduce
API admitting optionalheaders
Kafka
: TargetsFsKafka
v1.4.1
Propulsion.Kafka
/Kafka0
: ReplacedProduceAsync
API withProduce
that no longer yields aDeliveryReport
(this functionality can be replicated via an alternate overload)EventStore
: UpdatedCheckpointSeries
to adhere more closely to naming per standardized naming #51
2.1.1 - 2020-03-11
StreamsSyncStats
: Added missingHandleOk
StreamsProducerSink
: Added overload to facilitateprepare
make production of an outputoption
al and admitting processing of Stats #50
StreamsSync
: Add* 'outcome
tohandle
function signature #50- Update to
3.1.101
SDK - Retarget
netcoreapp2.1
apps tonetcoreapp3.1
(not least to make tool traverse proxies on Windows)
2.0.0 - 2020-02-19
- Target
FsCodec
.* v2.0.0
,Equinox
.* v2.0.0
2.0.0-rc3 - 2020-02-08
- Extend
StreamsProjector
overloads to enable simplified overriding of Write Position
2.0.0-rc2 - 2020-02-04
Kafka0
: Change"compression.type"
to"compression.codec"
to correctly supportConfluent.Kafka
v0.11.3
2.0.0-rc1 - 2020-01-31
- Target
FsCodec
.* v2.0.0
,Equinox
.* v2.0.0-rc9
#47 - Use
FsCodec.StreamName
to replace string names; remove need forcategorize
functions #47 - Updated
EventStore.Checkpoint
to adhere to standard naming/layout (Folds
->Fold
)
1.5.0 - 2019-12-06
Tool
: Change switches (without arguments) to upper caseKafka
: TargetsFsKafka
v1.3.0
Cosmos
: TargetsMicrosoft.Azure.DocumentDB.ChangeFeedProcessor
v2.2.8
EventStore
: TargetsTypeShape
v8.0.1
Tool
: UsesArgu
v6.0.0
- Update to
Microsoft.SourceLink.GitHub
v1.0.0
1.4.0 - 2019-11-14
- Targets
Equinox
v2.0.0-rc8
,FsCodec
v1.2.1
Cosmos
: Retarget to specify stores asDocumentClient
#40 🙏 @Kelvin4702
1.3.2 - 2019-11-13
StreamKeyEventSequencer
: Handlenull
keys #43 🙏 @nosmanEventStore.Checkpoint
: Fix to actually writeCheckpointed
events #44
1.3.1 - 2019-11-12
- Add
-g
toConsumerGroupName
for consistency with dotnet-templates#37
- EventStore: Handle
minBatchSize
< 128 - EventStore: Inhibit redundant checkpoint writing as originally intended in
1.0.1
1.3.0 - 2019-10-17
- Targets
Equinox
v2.0.0-rc7
,FsCodec
v1.0.0
- Updated MinVer to
2.0.0
1.2.1 - 2019-10-11
.Kafka.Core.StreamKeyEventSequencer
- helper to synthesize a stream index in the event of a source not providing it intrinsically.
.Kafka
: TargetedJet.ConfluentKafka.FSharp
v1.2.0
- rename
offsetCommitInterval
toautoCommitInterval
to match similar change inJet.ConfluentKafka.FSharp
v1.2.0
- Generalize
Checkpoints.Folds.transmute
to be directly usable #35 - Updated MinVer to
2.0.0-rc.1
NullReferenceException
when handlingnull
keys/values in Kafka messages #37 🙏 @jgardella
1.2.0 - 2019-09-15
- honored
maxBatchSize
in Kafka Consumers #31 offsetCommitInterval
toPropulsion.Kafka0
'sKafkaConsumerConfig
maximizeOffsetWriting
toStreamsConsumer
in order to give maximum effect tomaxBatchSize
limit implemented in #31- test suite for
StreamsConsumer
#32 BatchesConsumer
: support for custom stream based batch scheduling algorithms #29
- Use
IIndexedEvent
in lieu ofIIEvent
+index
inStreamSpan
andStreamEvent
#28 - Shorten
Rendered*.parse*
toRendered*.parse
#28 - Updated MinVer to
2.0.0-alpha.2
1.1.1 - 2019-09-07
- Targeted
Equinox
.* v2.0.0-rc6
,FsCodec
.* v1.0.0-rc2
#27
1.1.0 - 2019-08-30
- Rebased on
FcCodec
to enable cleaner interop withEquinox
. Includes removing redundant Converters and helpers #26 - Targeted
Equinox
.* v2.0.0-rc5
#26
1.0.1 - 2019-08-26
EventStore
: SwitchedCheckpoints
to correctly only log one event per hour using EquinoxRollingUnfolds
/transmute
mechanismKafka
/Kafka0
: AddedKafkaMonitor
based on Burrow #12 🙏 @jgardella- Added overloads,
Codec.RenderedSummary
andPropulsion.Streams.Sync
to supportdotnet new proSummaryProjector/Consumer
#23
- Targeted
Equinox
.* v2.0.0-rc3
#22 - Targeted
Equinox
.* v2.0.0-rc4
1.0.1-rc9 - 2019-08-09
Propulsion.Tool
:initAux
(nowinit
) andproject
facilities moved from Equinox #17Propulsion.Tool
:initAux
andproject
facilities moved from Equinox #17
- Targeted
Equinox
.* v2.0.0-rc2
- Resolve
Propulsion.Kafka0
conflicts withJet.ConfluentKafka.fsharp
v0 #19
1.0.1-rc8 - 2019-07-05
Kafka
/Kafka0
: Tuned stream producer params
1.0.1-rc7 - 2019-07-04
Kafka
: LogProducer
exceptionsKafka
:StreamsProducerSink
Enable control ofmaxEvents
,maxBytes
Propulsion
/Propulsion.Kafka
/Propulsion.Kafka0
: RemovedIEvent
onStreamEvent
,IEnumerable<IEvent>
onStreamEventSpan
1.0.1-rc6 - 2019-07-03
Kafka
/Kafka0
: RenameProducers
toProducer
, add deprecation not to [renamed]degreeOfParallelism
parameter
Kafka0
- added missing error check on produce #14
1.0.1-rc5 - 2019-07-02
Propulsion
: ImplementIEvent
onStreamEvent
,IEnumerable<IEvent>
onStreamEventSpan
Propulsion.Kafka
: ImplementedIEnumerable<IEvent>
onRenderedSpan
Propulsion.Kafka
: AddedParse
,parseStreamEvents
helpers toRenderedSpan
1.0.1-rc4 - 2019-07-01
.Kafka
: TargetedJet.ConfluentKafka.FSharp
v1.1.0
1.0.1-rc3 - 2019-06-18
Core
: Average Streams latency measurements/loggging forStreamsConsumer
#3Kafka
:customize
option forParallelConsumer
andStreamsConsumer
's.Create
methods #3Kafka
:producerParallelism
option #3Kafka0
: Provides source-compatibility withPropulsion.Kafka
targetingJet.ConfluentKafka.FSharp
v0.9.1
/Confluent.Kafka
v1.0.1
#4Kafka
/Kafka0
:Producers
- Common Kafka producer wrapper with metrics #9Kafka
/Kafka0
:StreamsConsumerStats
- Consumer outcome / statistics / logging support #9
Propulsion.Cosmos
: Tidied Cosmos ingester lag breakdown- Moved
RenderedSpan
et al toPropulsion.Codec.NewtonsoftJson
#5 - Targeted
Jet.ConfluentKafka.FSharp
v1.0.1
#3 - Targeted
Equinox
.* v2.0.0-rc1
#7
1.0.1-rc2 - 2019-06-07
Propulsion.EventStore.EventStoreSource
(productized fromEquinox.Templates
'seqxsync
) #1
- Targets
Microsoft.Azure.DocumentDB.ChangeFeedProcessor
v2.2.7
, which includes critical lease management improvements
1.0.1-rc1 - 2019-06-03
Propulsion.Kafka.Codec.RenderedSpan
(neeEquinox.Projection.Codec.RenderedSpan
, which is deprecated and is being removed)Propulsion.EventStore
,Propulsion.Cosmos
(productized fromEquinox.Templates
'seqxsync
andeqxprojector
)
- Targets
Jet.ConfluentKafka.FSharp
v1.0.1
1.0.0-rc13 - 2019-06-01
- Split reusable components of
ParallelConsumer
out into independentPropulsion
andPropulsion.Kafka
libraries #34
- Significant tuning / throughput improvements for
ParallelConsumer
ParallelConsumer
#33
(Stripped down repo for history purposes, see master
branch of Jet.ConfluentKafka.FSharp for complete history prior to 1.0.0-rc13)