Redis Cluster: Endpoint 10.42.241.4:6379 serving hashslot 1643 is not reachable at this point of time.

[CenturySparkle]

Using Redis clustered with 6 nodes, 3 master and 3 slave. In our production environment we sometimes get this error. About 1 or 2 a week.

Endpoint 10.42.241.4:6379 serving hashslot 1643 is not reachable at this point of time. Please check connectTimeout value. If it is low, try increasing it to give the ConnectionMultiplexer a chance to recover from the network disconnect.

The endpoint is up and running with many connections and no signs of having gone down. On average, all three masters are processing about 30-60 commands per second. The application and the Redis cluster are all running in Rancher Kubernetes within the same namespace.

[mgravell]

My first guess would be a server or network block - check SLOWLOG on the specific node to rule out the first; the latter is more dependent on data - huge single BLOBs etc can cause head-of-line blocking.