Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enhancement: persist commit index in LogStore to accelerate recovery #613

Open
wants to merge 37 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
2e5a8a0
feat: add CommitTrackingLogStore interface for commit index management
peterxcli Sep 1, 2024
ffc6b3b
chore: remove non-idiomatic type assert func
peterxcli Sep 3, 2024
7383d96
feat(raft): add fast recovery mode for quicker log application
peterxcli Sep 4, 2024
f6295e0
feat(raft): add recovery from committed logs during startup
peterxcli Sep 4, 2024
f2ae7a9
refactor(store): rename ReadCommitIndex to GetCommitIndex for consist…
peterxcli Sep 6, 2024
ce1895c
fix: also set inmem commit index when revocer log commit progress fro…
peterxcli Sep 10, 2024
ab50a58
perf: optimize startup recovery by skipping duplicated log replay
peterxcli Sep 10, 2024
4e7e04b
refactor(inmem-commit-tracking-store): store commit index in memory u…
peterxcli Sep 13, 2024
41df55e
chore: fix typo in recoverFromCommittedLogs function name
peterxcli Sep 13, 2024
400a27d
refactor(raft): update parameter name in persistCommitIndex function
peterxcli Sep 13, 2024
e2617e8
refactor(raft): set commit index in memory before `StoreLogs`
peterxcli Sep 13, 2024
6daca47
refactor(raft): fix condition for skipping recovery in `recoverFromCo…
peterxcli Sep 18, 2024
cc09317
feat(raft): add commit tracking logs and fast recovery tests
peterxcli Sep 18, 2024
fe57b32
docs(config): update comments for FastRecovery mechanism
peterxcli Sep 19, 2024
20e8701
refactor(inmem-commit-tracking-store): simplify in-mem log tracking s…
peterxcli Sep 19, 2024
6f146e1
fix: rename persistCommitIndex to tryPersistCommitIndex
peterxcli Sep 19, 2024
a8438b0
chore(raft): rename tryPersistCommitIndex to tryStageCommitIndex for …
peterxcli Sep 20, 2024
5e6d8a4
refactor(log): introduce StagCommitIndex for optimized atomic persist…
peterxcli Sep 20, 2024
e248f00
fix(raft): correct CommitTrackingLogStore implementation
peterxcli Sep 24, 2024
2a913ab
feat(raft): improve fast recovery error handling and commit index val…
peterxcli Sep 24, 2024
7cd6732
feat: add `CommitTrackingLogStore` interface check and adjust return …
peterxcli Oct 9, 2024
92c04a0
refactor: improve type assertion for log store in TestRaft_FastRecovery
peterxcli Oct 9, 2024
8e8ba07
feat: add warning log for unsupported fast recovery
peterxcli Oct 10, 2024
2020cab
refactor: move commitIndex retrieve into `tryStageCommitIndex`
peterxcli Oct 10, 2024
2a7d584
refactor: remove error from return field of recoverFromCommittedLogs
peterxcli Oct 10, 2024
bdac45b
refactor: rename FastRecovery and revert the stageCommittedIdx change
peterxcli Oct 11, 2024
ed47a25
docs: documented GetCommitIndex in CommitTrackingLogStore interface
peterxcli Oct 11, 2024
ad87d86
docs: change fastRecovery flag to recoverCommittedLog in all document…
peterxcli Oct 11, 2024
30fc43e
refactor: add a new ErrIncompatibleLogStore for recoverFromCommittedLogs
peterxcli Oct 11, 2024
e797962
docs: clarify RestoreCommittedLogs configuration requirement
peterxcli Oct 11, 2024
500567f
refactor: rename recoverFromCommittedLogs to restoreFromCommittedLogs
peterxcli Oct 11, 2024
cfffcb5
refactor!: update MakeCluster functions to return error
peterxcli Oct 11, 2024
560c0b9
test: add test for RestoreCommittedLogs with incompatible log store
peterxcli Oct 11, 2024
8c722fa
Revert "refactor!: update MakeCluster functions to return error"
peterxcli Oct 12, 2024
300a6e7
refactor: update makeCluster to return errors
peterxcli Oct 12, 2024
8d11a28
Use wrapped err
peterxcli Oct 12, 2024
1bdf161
docs: clarify GetCommitIndex behavior in CommitTrackingLogStore inter…
peterxcli Oct 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions log.go
Original file line number Diff line number Diff line change
Expand Up @@ -190,3 +190,13 @@ func emitLogStoreMetrics(s LogStore, prefix []string, interval time.Duration, st
}
}
}

type CommitTrackingLogStore interface {
SetCommitIndex(idx uint64) error
ReadCommitIndex() (uint64, error)
}

func isCommitTrackingLogStore(s LogStore) bool {
_, ok := s.(CommitTrackingLogStore)
return ok
}
lalalalatt marked this conversation as resolved.
Show resolved Hide resolved