Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allocator: research support for MemberSpec load factor #237

Open
jgraettinger opened this issue Nov 18, 2019 · 0 comments
Open

allocator: research support for MemberSpec load factor #237

jgraettinger opened this issue Nov 18, 2019 · 0 comments

Comments

@jgraettinger
Copy link
Contributor

As a Gazette operator, I'd like a dynamic control feedback loop which allows brokers having very busy journals to discharge those journals more equitably around the cluster.

Today the allocator (falsely) assumes all journals are utilized equally. As an operator, it's on me to partitionin journals to best approximate this assumption, but it's impractical to achieve perfectly. Journals may also exhibit shorter-lived write skew, eg due to time of day patterns.

I'd like to be able to determine a load factor for each broker (through a process not defined here, but likely kubernetes top or something similar), and then periodically update that factor on each broker's advertised MemberSpec. The allocator should then use that factor to determine the "fair-share" scaling for each broker, such that busy brokers discharge journals and idle brokers take on more.

I should have the means to automate this control system (eg, running every 5 minutes) to balance our broker load over the day.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant