Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Fetch Stream Out of Bound #1970

Closed
Chillax-0v0 opened this issue Aug 30, 2024 · 1 comment · Fixed by #1971
Closed

[BUG] Fetch Stream Out of Bound #1970

Chillax-0v0 opened this issue Aug 30, 2024 · 1 comment · Fixed by #1971
Assignees
Labels
bug Something isn't working

Comments

@Chillax-0v0
Copy link
Contributor

Version: 1.1.2-rc1

What's wrong:
image

What happened:
image
In the logs, we can see that a segment swap occurred before the issue happened. During this swap, the "log" segment was replaced with the "log.cleaned" segment, the latter of which used the problematic Stream-300979.

private void trimStream0(ElasticLogMeta meta) {
Map<String, Long> streamMinOffsets = new HashMap<>();
inflightCleanedSegments.forEach((offset, segment) -> {
ElasticStreamSegmentMeta segMeta = segment.meta();
calStreamsMinOffset(streamMinOffsets, segMeta);
});
for (ElasticStreamSegmentMeta segMeta : meta.getSegmentMetas()) {
calStreamsMinOffset(streamMinOffsets, segMeta);
}
streamManager.streams().forEach((streamName, stream) -> {
var minOffset = streamMinOffsets.get(streamName);
// if minOffset == null, then stream is not used by any segment, should trim it to end.
minOffset = Optional.ofNullable(minOffset).orElse(stream.nextOffset());
if (minOffset > stream.startOffset()) {
stream.trim(minOffset);
}
});
}

In the code, we can see that before trimming a stream, the minimum offset for each stream is calculated. Specifically, if a stream is not being used by any segment, it will be trimmed to the end.
However, when calculating the minimum offset, the inflightCleanedSegments (a member variable) and meta.getSegmentMetas() (a method parameter) do not represent the same view at the same point in time. This discrepancy can lead to the incorrect assumption that a stream is not being used by any segment.

@superhx
Copy link
Collaborator

superhx commented Sep 26, 2024

@superhx superhx closed this as completed Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants