You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
We can decouple rw_ddl_progress from meta’s materialized view progress tracker, and maintain it adhoc:
Make internal backfill state tables visible. They contain the backfilled row_count, and whether the backfill is finished or not.
a. Query all the internal backfill state tables for an MV to fetch row_count and finished status.
b. Query internal tables of the MV.
c. Regex the name for backfill tables.
Query the hummock version stats, so we can get the upstream row count.
Calculate the estimated progress.
This will:
Simplify logic of materialized view progress tracker. We no longer need to maintain state for counts.
Allow us to track the backfill progress of created sink jobs.
In general, I think the same pattern can be applied to other forms of backfilling, snapshot backfill and shared source backfill.
The text was updated successfully, but these errors were encountered:
We can decouple rw_ddl_progress from meta’s materialized view progress tracker, and maintain it adhoc:
a. Query all the internal backfill state tables for an MV to fetch row_count and finished status.
b. Query internal tables of the MV.
c. Regex the name for backfill tables.
This will:
Simplify logic of materialized view progress tracker. We no longer need to maintain state for counts.
Allow us to track the backfill progress of created sink jobs.
In general, I think the same pattern can be applied to other forms of backfilling, snapshot backfill and shared source backfill.
The text was updated successfully, but these errors were encountered: