FAQ

[gvolpe]

Q&A

Here's the place to ask general questions about the book, leave feedback, greetings, etc. If you have questions specific to a chapter, please refer to the chapter discussion instead.

FAQ

Where can I buy the digital version?

Single: https://leanpub.com/feda/
PFPS bundle: https://leanpub.com/b/pfp-feda-scala

Will there be a print version?

Yes, as soon as the digital version is 100% complete. Usually available on my Blurb account.

Where can I subscribe to updates?

Announcements will always be made in this discussion, you can subscribe there. Additionally, I usually make announcements on my Twitter account.

Book updates generally ship with release notes, which you can get via email, if you're subscribed to LeanPub updates.

[stoszkom]

I'm afraid this question is just about the code style.
What's the reason for creating an Acker from the Consumer?
Can't the Consumer just extend the Acker?
What about sth like:

trait Consumer[F[_], A] extends Acker[F, A] with Receiver[F, A]

trait Receiver[F[_], A]:
  def receiveM: Stream[F, Consumer.Msg[A]]
  def receiveM(id: Consumer.MsgId): Stream[F, Consumer.Msg[A]]
  def receive: Stream[F, A]

trait Acker[F[_], A]:
  def ack(id: Consumer.MsgId): F[Unit]
  def ack(ids: Set[Consumer.MsgId]): F[Unit]
  def nack(id: Consumer.MsgId): F[Unit]

I've seen sth similar here

[gvolpe]

trait Consumer[F[_], A] extends Acker[F, A] is definitely a good idea, thanks! See #79 👍🏽

Acker was a later addition, so I guess I missed the redundancy in those methods. Regarding Receiver, we don't have such a need in this application, but having that separate interface would also be valid if we did.

[stoszkom]

Thank you!

[stoszkom]

I'm struggling to understand how it is possible to run multiple instances of the processor service.
From what I understand:

• there is one TradeState
• all the instances read the same snapshot of the TradeState

How does the key-shared mode allow to modify this state so that there are no conflicts?
That's how I see it:

sequenceDiagram
participant S as Snapshots
participant P as Pulsar
participant P1 as Processor1
participant P2 as Processor2
participant R as Redis
par processing 1
P1 -->>+ P1: start
P1 -->>+ R: read snapshot
R -->>- P1: snaphot
Note right of P1: stateVersion=x
loop every cmd
P -->> P1: trade cmd1
P1 -->> P1: apply cmd1 to state
Note left of P1: stateVersion=x+y
P1 -->>- P: trade event1 (cmd1 executed)
end
and processing 2
P2 -->>+ P2: start
P2 -->>+ R: read snapshot
R -->>- P2: snaphot
Note right of P2: stateVersion=x
loop every cmd
P -->> P2: trade cmd2
P2 -->> P2: apply cmd2 to state
Note left of P2: stateVersion=x+z
P2 -->>- P: trade event2 (cmd2 executed)
end
and snapshot writer
loop every ticky | event
alt event in
P -->>+ S: trade event1
S -->> S: apply cmd1 to state
Note right of S: stateVersion=x+y
S -->>- P: ack
P -->>+ S: trade event2
S -->> S: apply cmd2 to state
Note right of S: stateVersion=x+y+z???
S -->>- P: ack
else tick in
S -->>+ S: tick
S -->>+ R: write snapshot
R -->>- S: ack
end
end
end

Loading

What do I get wrong?

[gvolpe]

@stoszkom these are good questions, and hopefully readers like you will find the answer in the book once it's finished. That said, neither the book nor the code was pair-reviewed yet, so early bugs are to be expected.

To partially answer your questions, let me start by adding some details to your understanding:

• there is one TradeState per processor instance.
• all the instances read the same snapshot of the latest TradeState.

The state is local to the processor instance, and it only lives in memory. On the other hand, the snapshots service is the one in charge of persisting the latest state (single writer principle) by listening to events.

Pulsar's KeyShared mode allows us to distribute messages via a hash that is configurable per message. In this case, we shard by symbol (see Shard instance), so two different instances of processor will never get a command for the same Symbol.

So even though two different processor instances can start from the same TradeState (from snapshots), their state transitions will only affect a specific symbol, which is never shared across different instances.

Thus, your understanding of the state transitions is correct, i.e. one getting x+y and another x+z, so I can see how that can be confusing, but know it doesn't affect the outcome thanks to the guarantee of always operating on different symbols.

Have a look at the SnapshotWriter too, where price updates are written per symbol, which are always processed by specific instances that don't get into conflict with one another.

That said, there still could be bugs I missed, and I actually found one in the Shard instance by looking at it. When sharding by symbol is not possible (meaning we either get Start or Stop), then we need to shard by "trading-status", so only one instance controls the switch. See #82

Hope that gives you some answers for now :)

[gvolpe]

While trying to explain this in detail in the book, I realized there could be an issue when consumers are rebalanced by Pulsar, so I'm currently working on fixing this potential issue. So thanks again for pushing me into thinking about more edge cases :)

[stoszkom]

I understand it now, thank you! It helps me a lot.
I'm currently learning about reactive programming.
Your book is one of the best and most accessible sources I've found out there.
It really gave me a wider understanding of the topic :)