You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As a Gazette operator, I'd like a dynamic control feedback loop which allows brokers having very busy journals to discharge those journals more equitably around the cluster.
Today the allocator (falsely) assumes all journals are utilized equally. As an operator, it's on me to partitionin journals to best approximate this assumption, but it's impractical to achieve perfectly. Journals may also exhibit shorter-lived write skew, eg due to time of day patterns.
I'd like to be able to determine a load factor for each broker (through a process not defined here, but likely kubernetes top or something similar), and then periodically update that factor on each broker's advertised MemberSpec. The allocator should then use that factor to determine the "fair-share" scaling for each broker, such that busy brokers discharge journals and idle brokers take on more.
I should have the means to automate this control system (eg, running every 5 minutes) to balance our broker load over the day.
The text was updated successfully, but these errors were encountered:
As a Gazette operator, I'd like a dynamic control feedback loop which allows brokers having very busy journals to discharge those journals more equitably around the cluster.
Today the allocator (falsely) assumes all journals are utilized equally. As an operator, it's on me to partitionin journals to best approximate this assumption, but it's impractical to achieve perfectly. Journals may also exhibit shorter-lived write skew, eg due to time of day patterns.
I'd like to be able to determine a load factor for each broker (through a process not defined here, but likely kubernetes top or something similar), and then periodically update that factor on each broker's advertised MemberSpec. The allocator should then use that factor to determine the "fair-share" scaling for each broker, such that busy brokers discharge journals and idle brokers take on more.
I should have the means to automate this control system (eg, running every 5 minutes) to balance our broker load over the day.
The text was updated successfully, but these errors were encountered: