Proposal: introduce dynamically scalable RAFT cluster.

[mzazaipsc]

Currently, dynamic membership changes in the RAFT cluster are possible using the addServer() and removeServer() methods of RaftHandle.
However, this is only possible if the member to be added is already present in the leader’s member list, requiring a consistent membership list before a node can join.

This proposal introduces dynamic joining and leaving of members.

The resulting interface would, for example, add join(String clusterName) and leave() methods to the RaftHandle class.
join() would try to communicate the request to the leader of the cluster, and only return successfully after the leader has added and committed the updated membership list (using addServer()), otherwise fail with an explicit cause (e.g. TimeoutException).

Similarly, leave() would initiate removal, but would be less complex since the node is already part of the cluster.
The application code must then call leave() before a node shuts down, to avoid negatively impacting quorum.

Internally, RaftHandle would need to introduce some communication mechanism for non-member nodes - e.g. through a dedicated messaging channel with an identical cluster name.

I have a working proof-of-concept that would benefit from further refinement, which can be taken forward if the proposal is accepted.

[jabolina]

Hey, @mzazaipsc, thanks!

By dynamically do you mean it would automatically react to view updates and submit commands to alter the Raft member list? You might want take a look at #176.

[mzazaipsc]

Hi @jabolina, thanks for the feedback!

I've read #176.

The idea is to allow a node (non-raft member node) to explicitly request adding itself to the raft cluster.

This would be a bit different by decoupling the join/leave processes from the raft view by sending requests through a non-raft medium.

For example, a node that is interested in becoming a raft member would act like this:

Sends a JoinRequest to the raft cluster expressing the will to join the cluster. Every raft member receives the request and it proceeds only if it was leader. The request must contain the RaftId the node wants to have when it has joined.

The leader processes the JoinRequest like this:

Adds the new node to the raft cluster using the already existing RaftHandle.addServer(request.raftId) by the leader.

If addServer succeeds, a response is sent back to the node, confirming that the member list has been updated and replicated successfully. The response contains the new member list.

Now the node has the member list of the cluster: it updates it's member list locally and connect - raftHandle.raft().members(response.members); raftChannel.connect().. Now it is in the raft view AND a member of the raft cluster!

The client-side (non-raft node) logic will be initiated by calling the proposed raftHandle.join() method (called before connecting to the cluster!)

If a success response was received, then raftChannel.connect() will be called by join(), if join() fails, it is up to the application logic to retry (ensuring correctness) or not (leaving the cluster in an incorrect state if the ID was added to the list by the sending of the response failed. More on this below).

Leaving will follow a very similar logic:

raftHandle.leave() will be called, issuing a LeaveRequest to the cluster to remove this member (again, raftId is included in the request). Request gets processed by the leader. Leader calls removeServer(request.raftId), after new member list is committed successfully, a response is sent back to the node, confirming the removal.

Join workflow: Non-raft member issues JoinRequest -> RaftCluster's leader adds the new node and commits the new member list (through removeServer()) -> Leader issues response -> Non-raft member has a complete member list (with itself included) can join (connect)!

Leave workflow: Raft member issues LeaveRequest -> RaftCluster's leader removes the node and commits the new member list -> Leader issues response -> raft member has been removed successfully -> free to leave!

Communication would go for example through an embedded messaging channel in the RaftHandle, so it doesn't require a node to be a raft-member to communicate with the raft cluster.

The idea of this proposal is not to introduce automatic membership change based on view change. It is on the contrary to untie the relationship between them in terms of joining and leaving and achieving dynamic change by forcing explicit requests made by the non-raft nodes.

Given that we would have here two views (one for sending requests/responses, and one of the raft cluster) it must result in: members.size() <= requestsChannelView.size() before join and members.size() <= requestsChannelView.size() after remove.

Given that everything goes through consensus, this can not affect quorum negatively.

The workflow by @belaban in #176 about the reloading of the membership list after restart can be considered:

• This requires that the members configuration is stored in/with the snapshot.
• When a new member starts, its initial membership is configured by the XML config file
• Then (if present) the snapshot is read and members changed
• Then the log is read, this may also contain configuration changes, changing members

Surely some concerns may arise regarding the non-atomicity of the join and leave operations (they both involve two steps on the leader side: addServer/removeServer and send response). This would be solved by the client-side (application logic) by keeping retrying until gets explicitly either accepted or rejected (i.e. the join() method returns without an exception). join() would start with a try to connect (raftChannel.connect()). So that it eventually either returns because it received a response or because it could connect because the raftId existed in the members list already

The description here is a bit oversimplified for brevity (not covering detailed steps, retires, failure handling, leader transition, leader self removal, and other edge cases), but it should deliver the main idea.

This in summary would:

• Allow new nodes to join without having any information about the RAFT cluster (or the cluster knowing about them) other than the cluster name. (Security can theoretically be integrated if needed)
• Allow the dynamic changes made to cluster's membership list to stay consistent after restarts.
• Explicitly hand over membership change requests to the application logic: leaving every thing obvious from the application developer point of view.
• Allow membership change to only happen through RAFT consensus.

[jabolina]

Great, @mzazaipsc, thanks for the details. Personally, I always find reconfiguration a bit of a pain and dangerous to handle correctly, that's why I asked about it being kind of automatic.

Now, about your proposal: this sounds like something I had in mind, and we can refine it further. I was hoping to add a role of LEARNER to the cluster, and this would be tied by membership changes (#354). I hope to work a little bit on it this week to come up with a design and investigate more, so I'll simplify a bit.

I addition to what I added in the learner issue about scaling, this would solve some issues with membership changes. I wanted to streamline this process and make it possible for other nodes, outside of the Raft membership, to join the cluster. This is in line with your proposal, but instead of exposing the APIs, I wanted to add it as a new role.

A learner node would be any node inside the JGroups view but outside the Raft membership. That is, a raft_members={A, B, C} with a view={A, B, C, D}, will have node D with the role of LEARNER. This node would be able to join the cluster by connecting to the JChannel, and it would receive the replicated commands from the leader, just as a regular follower. The main difference is that learners do not vote for replicating entries, do not vote for leader election, and do not become leaders. They only learn about what was replicated in the cluster and apply it to their state machine.

The step similar to your "join" (and subsequently, the "leave") workflow is that it only includes the membership changes submitted by the addServer/removeServer API. If there is a leader and a majority available, the membership command is committed, and the learner will also apply it locally. After the node is included in the Raft list, it will automatically move from LEARNER to FOLLOWER. The application could invoke the addServer/removeServer when it wants the node to become part of the cluster as a follower.

Also, note that we have a REDIRECT protocol in the stack. This should automatically redirect the command to the current leader, including membership changes.

Please, let me know what you think. I believe it also has the same end goals as your proposal. I hope to spend some time this week to come up with a proper design document for learners.

[belaban]

Nice idea to expose an API for adding / removing nodes to the core cluster. I also like that learners and participants/leaders share the same JGroups cluster; I wouldn't like to have to create a separate cluster just for these JOIN requests.

[mzazaipsc]

Thank you for the detailed feedback. I've read #354 already, but the explanation here about the implementation gave me a broader idea of how this would be achieved. I agree with you that my proposal really overlaps in terms of goals with your explanation.

So to be on the same page: the new model suggests that any node after joining the JChannel automatically starts as a learner, and hence able to communicate with the cluster and redirect write commands to the leader. This would mean:

• addServer and removeServer would be callable from any raft handle (now only by leader, exception otherwise)

• connect() call on the raft channel would be allowed by any node that knows the cluster name? (now only possible for known members).

This would indeed enable dynamic members change.

One more addition I can think of this is a promotion policy, so that a learner might be promoted:

• Eventually by the leader when the learner has caught up -> optimal for not overwhelming the leader, preventing "an availability issue (that) can occur if many new
  servers are added to a cluster in quick succession" according to Raft dissertation 4.2.1.

• Or by application code to decide when to change a node's role (redirect command to leader).

—————————

Now to the overlap of both approaches:

While I think the automatic joining of any node as a raft member is a very nice idea, I can think of some questions:

• Would this affect the election mechanism? (given view changes).

• Wouldn't this, additionally, introduce some tight coupling between the channel view and the membership?

• And how achievable is this with only depending on the current API (with only changes to the implementation)?

• Would this introduce two sources of truth for members (view for learners and membership list for followers/leader).

To address those points, I would suggest an integration path where both ideas (learner role and join/leave model) can fit seamlessly:

Joining nodes would benefit from learner role in terms of performance and availability, while still providing separation of concerns and isolation between members and non-members, and avoiding automatic membership change.

I suggest the following:

Nodes can be added to the cluster explicitly using the proposed join/leave API, with two options/intents that align with the aforementioned promotion policy (more on these below) this would:

• Make it less complex and more explicit to add nodes to the cluster. (No automatic change)
• Keep the coupling between view changes and membership changes loose.
• Keep intents clear for the API caller, with two options for adding members:

1. Add a follower (starts as a learner and then eventually promoted by leader as soon as he catches up, Raft 4.2.1) - this would be the default. Starting as learner would result in performance, scalability and availability gains.
2. Add a learner that should not be promoted by the leader automatically, rather later by application code (e.g. using an API call).
3. (Optional) immediate add: node becomes immediately a follower without going through the learner phase.

• Keep joining/leaving only go through RAFT consensus.
• Work out of the box with current APIs (addServer, removeServer, connect() stay unchanged).
• Members are differentiated only by their roles, not by network topology or view.

Here we only need to figure out a way to let non-members communicate their join requests to the raft cluster (given the dedicated channel is not preferable by @belaban).

As a result, this integration would enable dynamic change of the raft membership list with the join/leave API, explicitly by the application code, while taking advantage of the learner role in terms of performance and availability. So that new members go through the learner phase, as soon as they join, continuing their lifecycle according to promotion policy.

These are quick ideas that came to my mind, so they surely need some refinement and detailing. It would be great if you share more ideas/thoughts on this. Thanks for the great discussion!

[jabolina]

Great, thanks, @mzazaipsc! I'll be on a public holiday tomorrow, so I hope to get a design document by the end of the week. This way, we have something more concrete to discuss. Likely, we can also do it step-by-step, we start with something simpler, but to hold as a foundation, and we can build any extra APIs we see lacking.

I'll open a PR once I have everything in place and would greatly appreciate comments.

[jabolina]

Hey, @mzazaipsc! Just opened: #356.