Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decouple show jobs from CreateMviewProgressTracker #19189

Open
kwannoel opened this issue Oct 30, 2024 · 1 comment
Open

Decouple show jobs from CreateMviewProgressTracker #19189

kwannoel opened this issue Oct 30, 2024 · 1 comment
Assignees
Milestone

Comments

@kwannoel
Copy link
Contributor

We can decouple rw_ddl_progress from meta’s materialized view progress tracker, and maintain it adhoc:

  1. Make internal backfill state tables visible. They contain the backfilled row_count, and whether the backfill is finished or not.
    a. Query all the internal backfill state tables for an MV to fetch row_count and finished status.
    b. Query internal tables of the MV.
    c. Regex the name for backfill tables.
  2. Query the hummock version stats, so we can get the upstream row count.
  3. Calculate the estimated progress.

This will:
Simplify logic of materialized view progress tracker. We no longer need to maintain state for counts.
Allow us to track the backfill progress of created sink jobs.
In general, I think the same pattern can be applied to other forms of backfilling, snapshot backfill and shared source backfill.

@github-actions github-actions bot added this to the release-2.2 milestone Oct 30, 2024
@kwannoel
Copy link
Contributor Author

Implementation-wise, we need to consider querying permissions for the internal table.

@kwannoel kwannoel self-assigned this Oct 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant