Horde.NodeListener.set_members times out when adding many nodes at once

[aloukissas]

I'm running my app on k8s and I notice when I bump the number of pods from e.g. 4 to 20, the call to Horde.NodeListener.set_members times out with the default GenServer.call/3 timeout of 5000 msec. This in turn kills the server and causes the pod to recycle, which starts a vicious cycle. I currently avoid this by slowly increasing the number of pods (e.g. by 2 at a time).

[keatz55]

@aloukissas We are running into a similar issue. Did you find a workaround or do you still have your k8s rollout limited to 2 pods at a time?

[derekkraan]

There's a fix coming for this (eventually). I need to make it possible (in DeltaCRDT) to make multiple mutations in one call; the problem right now is that it's having to add each new host one by one, which takes too long if you're going from 2 to 20 nodes.

[derekkraan]

Hi @aloukissas and @keatz55 can you please test with 0.8.4 (just released) and let me know if this fixes the problems you're facing with bumping number of nodes suddenly? Registry performance should also be improved (when nodes go down).

[RealVidy]

@derekkraan I have just tested with 0.8.4 and I get this timeout when starting 2 nodes concurrently on a slow machine.

Setting the timeout value to :infinity did "solve" the issue, of course.