tackle open segment start index ambiguity

[MathieuBordere]

There have been some discussions and issues regarding how raft determines the start index of an open segment.

The issue first popped up in canonical/raft#264 where the concept of a non-blocking barrier was introduced. This allowed a snapshot to run with a barrier while still allowing writes to new segments. The advantage of this approach was that all open segments were closed before taking a snapshot, and that the open segments in the data folder are always newer than the snapshot, leading to less/no ambiguity when determining their start index. The disadvantage of canonical/raft#264 is that there are now 2 types of barriers, that interact and that certain bugs popped up when running multiple barriers simultaneously (like canonical/raft#435).

canonical/raft#446 was an attempt to remove the concept of a non-blocking barrier, but it would again lead to ambiguity determining the start index of an open segment, because a snapshot can truncate away all closed segments in the data folder. At startup, raft does not know if the segments are older than the snapshot (in the snapshot take scenario) or newer than the snapshot (in the snapshot install scenario). Disambiguating these scenarios would require adding more info to the data folder or introducing special rules when truncating the log like

Don't truncate away the last closed segment, or we can't calculate the start index of an open segment anymore.

The better solution would probably be to encode the start index unambiguously in the segment, canonical/raft#444 is an attempt to do this. Implementing this would break forwards compatibility.

This issue is a place to discuss how to move forward, the goal would be to:

a) remove the 2 types of barriers, so we are left with only the blocking variant.
b) remove ambiguity when calculating the start index of an open segment.