Dynomite cluster does not respond when peer node is stopped

[ghost]

Hi
I build the dynomite cluster in datacenter1 and in rack1 with 3 nodes . I got the issue peers not connected while I performing SET/GET operation, if one of the node is stopped . please give the solution How can I will make it work , eventhough one of the node is stopped

[ghost]

Please give the solution as early as possible

[smukil]

@chinmayvenkat Sorry for the slow response. Can you paste the actual error message? Also, can you paste your conf files?

[leonardodavinte]

Hello brother,

I'm having the same problem. If my node01 is out of service I get the error:
NODE 01 Down
"127.0.0.1:8102> get KEYTEST
(error) ERR Peer: Peer Node is not connected "

But if my node02 and node03 are out of service, this does not impact the service.

NODE 02 or NODE 03 Down - OK
127.0.0.1:8102> get KEYTEST
"TESTE01"

I'm still learning about Dynomite, but apparently Node01 is a kind of coordinator of the Rack cluster nodes.

It seems to me a single point of failure. Any node can be out of service, except on the node coordinator, in this case node01.

[smukil]

@leonardodavinte This is because you are specifically talking to Node 1 using the redis-cli. i.e. 127.0.0.1:8102 is your node 1's address:port. If you connect to Node 2 or Node 3's address:port, you will still be able to use the cluster.

If you want a client that can automatically detect node failures and fallback to a different node, you will need to use our Dyno Java client:
https://github.com/Netflix/dyno

[ghost]

Hi which property we need to set for it. is dyno client supports single ton mapping . I am using pool. please give any examples. Thanks in advance

…

On Tue, Mar 26, 2019, 9:13 PM Sailesh Mukil ***@***.***> wrote: @leonardodavinte <https://github.com/leonardodavinte> This is because you are specifically talking to Node 1 using the redis-cli. i.e. 127.0.0.1:8102 is your node 1's address:port. If you connect to Node 2 or Node 3's address:port, you will still be able to use the cluster. If you want a client that can automatically detect node failures and fallback to a different node, you will need to use our Dyno Java client: https://github.com/Netflix/dyno — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#631 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AO65gYDQ_ykE5Rr-O4DIsGDniFj29sfrks5vakAmgaJpZM4afbgY> .

[smukil]

@chinmayvenkat A property for what specifically? There is a demo implementation for Dyno here. Hopefully that helps:
https://github.com/Netflix/dyno/blob/master/dyno-demo/src/main/java/com/netflix/dyno/demo/redis/DynoJedisDemo.java

[ghost]

I implemented singleton implementation model for dyno client. if one of the node cluster was down. it is giving errors. if we re established the connection when the node get down. it is working fine. How can we handle this?

…

On Tue, Mar 26, 2019, 9:21 PM Sailesh Mukil ***@***.***> wrote: @chinmayvenkat <https://github.com/chinmayvenkat> A property for what specifically? There is a demo implementation for Dyno here. Hopefully that helps: https://github.com/Netflix/dyno/blob/master/dyno-demo/src/main/java/com/netflix/dyno/demo/redis/DynoJedisDemo.java — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#631 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AO65gVRXF4taiaidzgQNlsSdoYeT5-c7ks5vakH8gaJpZM4afbgY> .

[leonardodavinte]

@leonardodavinte This is because you are specifically talking to Node 1 using the redis-cli. i.e. 127.0.0.1:8102 is your node 1's address:port. If you connect to Node 2 or Node 3's address:port, you will still be able to use the cluster.

If you want a client that can automatically detect node failures and fallback to a different node, you will need to use our Dyno Java client:
https://github.com/Netflix/dyno

Hello @smukil

Not exactly. I turned off node 01, and I try to connect to node 02 or node 03.

But with node 01 turned off, both node02 and node03 have errors:
"127.0.0.1:8102> get KEYTEST
(error) ERR Peer: Peer Node is not connected "

In this case, I'm connected via localhost on node02 and getting the error "Node is not connected".

Do you have any examples to configure
1 Datacenter
1 Rack
3 Nodes
?

Example:
DC1
RACK1
node01 - 192.168.0.1
node02 - 192.168.0.2
node03 - 192.168.0.3

How would you do it?

Thank you so much.

[smukil]

@chinmayvenkat How many replicas do you have in your configuration?

[smukil]

@leonardodavinte Ah my bad, I misunderstood your previous post. So based on your configuration, you have only 1 replica for each key. And you have 3 shards.

So, this means that the key KEYTEST probably lives in the Node01 shard, and since that node is down, it cannot access that key. If you add 2 more racks, then you should be able to fallback to another replica even if Node01 is down in one rack.

[leonardodavinte]

@leonardodavinte Ah my bad, I misunderstood your previous post. So based on your configuration, you have only 1 replica for each key. And you have 3 shards.

So, this means that the key KEYTEST probably lives in the Node01 shard, and since that node is down, it cannot access that key. If you add 2 more racks, then you should be able to fallback to another replica even if Node01 is down in one rack.

@smukil Thanks for the explanation.

[ghost]

@smukil 3 replicas I have
I had 3 racks consists of two servers in each rack
read and write consistency levels are dc_quorum

[ghost]

@smukil can you please tell me. how to see how many read/requests coming to dynomite per second
I used curl http://localhost:22222/info but I cannot understand how to see

[smukil]

@chinmayvenkat If you're using the Dyno client, then it should automatically fallback to a different replica. Have you made sure that your configuration is correct? You can check your topology with:
http://localhost:22222/cluster_describe
Also make sure that your HostSupplier has the right hosts:
https://github.com/Netflix/dyno/wiki/Configuration#hostsupplier

http://localhost:22222/info does have the right information:
... "client_connections":1, "client_read_requests":0, "client_write_requests":4, ...

[ghost]

@smukil . yes dyno client is handling the things. but it is not worked when I implement it for dyno in singleton implementation. please go through
Netflix/dyno#258
@smukil please give response to the
#644
as early as possible. it is painful point to me

[smukil]

Closing this, addressed the comment in:
Netflix/dyno#258 and #644