Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Scale-in nodes failover compact the StreamSetObjects #706

Open
superhx opened this issue Jan 23, 2024 · 1 comment
Open

Scale-in nodes failover compact the StreamSetObjects #706

superhx opened this issue Jan 23, 2024 · 1 comment
Assignees
@superhx
Labels
enhancement New feature or request Stale

Comments

@superhx
Copy link
Collaborator

superhx commented Jan 23, 2024

Who is this for and what problem do they have today?

Why is solving this problem impactful?

Compact the StreamSetObjects of the scale-in nodes with splitThreshold=1MiB :

  • Most of the data: Most of the data is split into StreamObjects to achieve independent TTL and better data cohesiveness.
  • Remaining 5000 partitions * 1MB: The remaining data will be periodically scanned and eventually expire through StreamControlManager#cleanupScaleInNodes.

Additional notes
@superhx superhx added the enhancement New feature or request label Jan 23, 2024
@superhx superhx self-assigned this Jan 23, 2024
@superhx
Copy link
Collaborator Author

superhx commented Jan 23, 2024 •
edited
Loading

We need to prepare more CPU / Memory resources for compaction failover.

https://github.com/AutoMQ/automq-for-rocketmq/tree/kafka_issues706

Implement it when you have a more elegant solution.

@github-actions github-actions bot added the Stale label Apr 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@superhx superhx

Labels
enhancement New feature or request Stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@superhx