Data loss in redis stream pubsub

[fsedano]

Expected Behavior

When using a dapr pubsub component, I would expect it to:

• Respect resiliency policies
• Not having data loss in the event of a app pod restart

Actual Behavior

To be able to set policies etc, we need to set processingTimeout and redeliverInterval to zero. This works well, but causes message loss if our pod restarts, since it seems data is pulled from redis and stored in the sidecar.

So, either we set them to zero and face data loss if the pod restarts, or set them to non-zero and we have to implement policies in the app side, making dapr useless for us.

Steps to Reproduce the Problem

• Create a redis streams pubsub, using k8s
• Create a k8s deployment with a health check
• Set processingTimeout=0 and redeliverInterval=0
• Start publishing and consuming messages successfully
• Simulate a failure: Return RETRY for some of the messages and after some time, make the app unhealthy. Make sure k8s health checks catches it and restart the pod.
• When the pod comes online again, the messages app returned as RETRY won't be redelivered again, they are lost.

Release Note

RELEASE NOTE:

[atrauzzi]

Hey @fsedano! We went through and did some checks on our end:

• With metadata settings processingTimeout=0 and redeliverInterval=0, we don't expect to see any retries of failures. This is a design decision as Dapr will not look for any PENDING messages on the Redis consumer group when these two are set.
  • This is separate to messages where the application returns RETRY. By explicitly returning RETRY this overrides the Dapr/component metadata settings and consistently retries the call forever. This is also expected behaviour.
• For scenarios where processingTimeout and redeliverInterval are non-zero, and redisMaxRetries is non-zero, we did notice a bug where the maxRetries value was being exceeded.

If the maxRetries value is respected would this solve your problem?