Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support scaling of backfill jobs #19142

Open
Tracked by #16612
kwannoel opened this issue Oct 27, 2024 · 3 comments
Open
Tracked by #16612

Support scaling of backfill jobs #19142

kwannoel opened this issue Oct 27, 2024 · 3 comments
Assignees
Milestone

Comments

@kwannoel
Copy link
Contributor

kwannoel commented Oct 27, 2024

We don't support scaling backfill jobs at the moment. So when we scale backfill jobs running under background ddl, they will trigger an assertion panic as mentioned by #19138.

This is prerequisite of serverless backfill and for improved ux of background_ddl, so we should support it.

@github-actions github-actions bot added this to the release-2.2 milestone Oct 27, 2024
@kwannoel kwannoel changed the title Support Background ddl scaling Support scaling of backfill jobs Oct 27, 2024
@kwannoel
Copy link
Contributor Author

kwannoel commented Oct 27, 2024

cc @shanicky, in case you already have ideas about supporting it, or have already started work on it before, so we avoid double work on it.

@xxchan
Copy link
Member

xxchan commented Oct 30, 2024

This is a little surprising, since I thought it's already supported.
I thought we disallow foreground job to be scaled, but allow background ddls to be scaled.

@kwannoel
Copy link
Contributor Author

This is a little surprising, since I thought it's already supported. I thought we disallow foreground job to be scaled, but allow background ddls to be scaled.

Unfortunately not. That should be supported, but at that point in time, several logic needs to be changed in the etcd backend and the sql backend, and background_ddl wasn't too high priority. So it's deferred to now.

Priority is increased since the logic will possibly be used for serverless backfill as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants