-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Redis Cluster not recognizing new cluster nodes with identical hostnames #1778
Comments
annervisser
added a commit
to annervisser/ioredis
that referenced
this issue
Jun 27, 2023
… connections when nodeId changes Fixes redis#1778
annervisser
added a commit
to annervisser/ioredis
that referenced
this issue
Jun 27, 2023
… connections when nodeId changes Fixes redis#1778
annervisser
added a commit
to annervisser/ioredis
that referenced
this issue
Jun 27, 2023
… connections when nodeId changes Fixes redis#1778
annervisser
added a commit
to annervisser/ioredis
that referenced
this issue
Jun 27, 2023
… connections when nodeId changes Fixes redis#1778
annervisser
added a commit
to annervisser/ioredis
that referenced
this issue
Jun 27, 2023
… connections when nodeId changes Fixes redis#1778
annervisser
added a commit
to annervisser/ioredis
that referenced
this issue
Jun 27, 2023
… connections when nodeId changes Fixes redis#1778
annervisser
added a commit
to ScientaNL/ioredis
that referenced
this issue
Jun 28, 2023
… connections when nodeId changes Fixes redis#1778
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We're in the process of testing MemoryDB and it's failure cases. We've run into an issue when upgrading a cluster while our application is using ioredis.
Environment
Problem
When changing the instance type of a MemoryDB cluster, an in-place upgrade happens where new nodes are deployed with the same hostnames as the old ones. The old ones start returning
MOVED
to all queries, and after a while they are removed.The way ioredis reacts to this is it ends up in a loop of retrying commands and receiving
MOVED
. This loop is only broken once the old nodes are removed and the individual connections time out.Cause
The
MOVED
responses cause ioredis to refresh it's slots, which ends up resetting the connectionpool with the new slots:ioredis/lib/cluster/index.ts
Line 902 in 96a9f41
But because the hostname and port are identical to the old nodes1, no new connections are made and the old ones are left intact. 2
Solution
I believe this could be fixed by including the nodeId (the third return value from
CLUSTER SLOTS
3) in the node keys, so that a new id will trigger a new connection.Footnotes
https://github.com/redis/ioredis/blob/d1d9dba9eafc574a9d9041fd4bc7cd04f1584159/lib/cluster/util.ts#L25-L29 ↩
https://github.com/redis/ioredis/blob/06782e681500eae6f3ceafcc6385b9be4fdaf4e3/lib/cluster/ConnectionPool.ts#L125-L135 ↩
https://redis.io/commands/cluster-slots/ ↩
The text was updated successfully, but these errors were encountered: