Connection instability issues bubbling up

[AaronMousaviSitecore]

Hi all,

Ever since our self-hosted Redis instances moved to Azure Managed Redis (AMR) we have been seeing a lot of exceptions coming from StackExchange.Redis which weren't there before. AMR has a lot more moving parts compared to the previous set-up and that seems to cause more connections getting dropped and replaced. The issues are intermittent and StackExchange.Redis does a good job of restoring the connections, but in the meantime it is bubbling up exceptions that are causing issues.

Could you please handle these exceptions internally? This avoids a lot of extra work on our side, like adding a retry on many places and it keeps the code cleaner. StackExchange.Redis is just fine with handling these issues, but while it is handling the issue, can it e.g. wait or retry the operations internally?

Some details of our set-up:

• StackExchange.Redis 2.8.31
• .NET 8.0
• Ubuntu 22.04.5 LTS
• Kernel 5.15.0-1092-azure x64

Some (redacted) exceptions:

StackExchange.Redis.RedisConnectionException: SocketClosed on instance_name.australiaeast.redis.azure.net:10000/Subscription, Idle/MarkProcessed, last: UNSUBSCRIBE, origin: ReadFromPipe, outstanding: 1, last-read: 0s ago, last-write: 0s ago, unanswered-write: 0s ago, keep-alive: 60s, state: ConnectedEstablished, mgr: 9 of 10 available, in: 0, last-heartbeat: 0s ago, last-mbeat: 0s ago, global: 0s ago, v: 2.8.31.52602
   at StackExchange.Redis.ChannelMessageQueue.UnsubscribeAsyncImpl(Exception error, CommandFlags flags) in /_/src/StackExchange.Redis/ChannelMessageQueue.cs:line 351

System.InvalidOperationException: Reading is not allowed after reader was completed.
   at System.IO.Pipelines.ThrowHelper.ThrowInvalidOperationException_NoReadingAllowed()
   at StackExchange.Redis.PhysicalConnection.ReadFromPipe()

The problem is similar to the ticket here #2692. It is very unlikely this relates to dispose issues as we have not seen that in years and the related code has not changed. The only change was moving to AMR (which including moving from Redis 6 to 7).

[mjgaux]

OK; so, a few thoughts in random order (based in part on discussions with Nick and Philo)

1. the exceptions themselves are fundamentally trying to say "something didn't work, unexpectedly"; we need to do/report something (if it isn't fire-and-forget)
2. some of the exceptions are perhaps too detailed, giving too specific details; maybe we should simplify this to a higher level / simpler exception, with the detail as the InnerException - you probably don't care about things like readers and sockets; do you have more examples of offending error types / messages?
3. for why you're seeing this more on AMR: AMR is currently lacking "maintenance notification events" - we're working on this between the library (SE.Redis), service (AMR), and server (Redis) teams - this should improve hopefully early in the new year

[mgravell]

(sorry, multiple GitHub accounts; ^^^ @mjgaux is also me @mgravell )

[AaronMousaviSitecore]

Thanks for the input @mgravell!

It's good to know about those maintenance notification events. Makes sense that that would improve things.

What do you recommend us to do? Put retry logic where we encounter issues? If yes, any pattern that is recommended?

About the exceptions, that could help. Some information about what might be underlying issue would be nice. What could potentially also help is if these exceptions all had a base type specific to StackExchange.Redis and maybe a property that signals whether a retry on the consuming side is recommended or not. This helps us to filter out StackExchange.Redis specific exceptions. Right now we are checking whether the exception message is Reading is not allowed after reader was completed.

I am checking for more exceptions in the meantime.