Will SmartStream provide Kafka Streams like capabilities?

[oronsh]

Hello,
First of all, let me say, HUGE thank you for this amazing project!
It feels like you really understood and nailed the problem with data processing :)
I'm very interested in the SmartStream feature which supposed to be the answer to Kafka Streams I guess?
I have several questions please:

1. Is smart streams going to provide similar features of kafka streams, such as windowing aggregations?
2. Why did you choose SmartStreams to be server side? wouldn't it be more efficient to use as a streaming library at the client side like kafka streams? This way it could be just pure rust rather than wasm too.

Thanks a lot!

[sehz]

Thanks for the compliments.

Regarding questions,

1. Yes, we are planning to add time and windowing aggregations.
2. There are a few benefits of doing server-side streaming computation. First, it minimizes data movement since data doesn't have to go outside the cluster unnecessarily. Moving data out of a cluster can be expensive, depending on cluster deployment and network topology. Second, a particular streaming operation can only be performed inline (for example, stripping out PII data) because of privacy and security concerns.

However, there are specific scenarios where it makes sense to perform processing on the client-side. To support this, we are planning to support processing anywhere, including the client-side. What use cases do you have in mind for client-side processing?

[oronsh]

Thanks for your answers.
I'm not sure how you implement aggregations today but for example, kafka streams aggregations are stored in a separated storage layer like rocksdb which would allow a persisted state. When done client side, that state is distributed, I know it has its own problems and tradeoffs. I'm just imagining a use case where most of your consumers need to aggregate data, so keeping all of the "aggregated state" in server side seems like an overhead to me since the cluster is responsible for holding the state rather each client holding its own relevant state, especially with high cardinality.

[sehz]

Can you explain about high cardinality state? Both Flink and Spark performs aggregation in their cluster

[oronsh]

Yes, for example at the company I work for we match structured payloads in kafka streams. For example:
payload {host: "xyz"}, now we do windowing aggregations and we ask, does host had the value xyz in the last 5 hours?
where host can already have million values aggregated in rocksdb in this time window.
I'm pretty new to stream processing but was just wondering if cluster aggregations could handle such large states in a multi tenancy fashion.

[sehz]

Definitely. We are designing for very large scales with real-time performances.

[oronsh]

Sounds amazing to me, can't wait :)