-
Notifications
You must be signed in to change notification settings - Fork 431
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: introduce LogStore
trait
#1706
Closed
Closed
Changes from 5 commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
2f475e2
refactor: introduce `LogStore` trait
dispanser 6330d8b
Merge branch 'main' into logstore-trait
rtyler db33e9e
Merge branch 'main' into logstore-trait
rtyler 84e4659
Merge branch 'main' into logstore-trait
dispanser b54c716
Merge branch 'main' into logstore-trait
dispanser 7922491
Remove `overwrite` flag for write
dispanser 748e050
Merge branch 'main' into logstore-trait
dispanser File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
//! Delta log store. | ||
use crate::errors::DeltaResult; | ||
use bytes::Bytes; | ||
|
||
use crate::protocol::Action; | ||
|
||
/// Trait for critical operations required to read and write commit entries in Delta logs. | ||
/// | ||
/// The correctness is predicated on the atomicity and durability guarantees of | ||
/// the implementation of this interface. Specifically, | ||
/// | ||
/// - Atomic visibility: Any commit created via `write_commit_entry` must become visible atomically. | ||
/// - Mutual exclusion: Only one writer must be able to create a commit for a specific version. | ||
/// - Consistent listing: Once a commit entry for version `v` has been written, any future call to | ||
/// `get_latest_version` must return a version >= `v`, i.e. the underlying file system entry must | ||
/// become visible immediately. | ||
#[async_trait::async_trait] | ||
pub trait LogStore { | ||
/// Read data for commit entry with the given version. | ||
/// TODO: return the actual commit data, i.e. Vec<Action>, instead? | ||
async fn read_commit_entry(&self, version: i64) -> DeltaResult<Bytes>; | ||
|
||
/// Write list of actions as delta commit entry for given version. | ||
/// | ||
/// This operation can be retried with a higher version in case the write | ||
/// fails with `TransactionError::VersionAlreadyExists`. | ||
async fn write_commit_entry( | ||
&self, | ||
version: i64, | ||
actions: Vec<Action>, | ||
overwrite: bool, | ||
) -> DeltaResult<()>; | ||
|
||
/// Find latest version currently stored in the delta log. | ||
async fn get_latest_version(&self) -> DeltaResult<i64>; | ||
} |
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What does overwrite do? Why is it necessary?(I would think overwriting a commit would be not compatible with consistency.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Great point. I copied that over from
LogStore::write
from the reference implementation, but couldn't trace back any justification for theoverwrite
flag. The design doc linked above has the following to say:which doesn't help me understand the "why" either. As you correctly point out, overwrite is violating any of the consistency the delta log is supposed to deliver in the first place.
If you don't see a use case / call to that method that would set
overwrite = true
, my proposal would be to drop that argument fromwrite_commit_entry
and potentially add it when we can justify its existence.