Local effects (send_msg, log etc) can be duplicated when adding a new member

[kjnilsson]

Describe the bug

Local effects will be executed by the leader if at the time of processing the effect the target node does not have a current member on it. In the case where a new member is added to a node and the log currently command that would generate local effects for this node the new member will re-execute all these effects as local.

E.g. in a RabbitMQ cluster of A, B, C where there is a quroum queue with a leader on A and a follower on B. If there is a consumer on C that has received some messages (sent by leader) and these have not yet been truncated from the log if a member is added on C whilst this consuming channel is still alive the new member on C will send all the already sent messages in the log to the consumer. (In RabbitMQs case the consuming code will just filter these messages out).

This means catching up could have a performance penalty (of processing all these local effects) which means it may take longer for the new member to catch up.

Reproduction steps

See above

Expected behavior

We need to find a given "pivot" point where a member starts executing local effects. It may be enough to check that the new member is in the cluster configuration but if a member with the same name is added and removed then added again this could still result in duplication which may or may not be acceptable.

Alt we could use the voter status in #375 as the pivot for whether a leader or local member executes the effect or not.

Additional context

No response

[illotum]

For a generic replicated state machine, the non-voters do not have up to date state in order to execute commands faithfully. Agree that it would be nice to at least wait for the follower to catch up.