-
Notifications
You must be signed in to change notification settings - Fork 133
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Parallel DEDUP_SEMI and DEDUP_SEMI_REVERSE Join.(A new approach to process Semi Join Query in Parallel of MPP) #653
Conversation
f32844c
to
926ee79
Compare
Another flaky failed cases of resgroup: |
Nestloop tends to use normal plan instead of Parallel, need a dig. |
7d5b18e
to
511df86
Compare
To handle semi join in parallel, we have enabled Parallel Semi join. This commit enable DEDUP_SEMI and DEDUP_SEMI_REVERSE join in parallel to handle semi join cases which could be a win in MPP mode. Both parallel-oblivious and parallel-aware are enabled, for an example of parallel-aware case: select * from foo where exists (select 1 from bar where foo.a = bar.b); QUERY PLAN --------------------------------------------------------------------------- Gather Motion 6:1 (slice1; segments: 6) -> HashAggregate Group Key: (RowIdExpr) -> Redistribute Motion 6:6 (slice2; segments: 6) Hash Key: (RowIdExpr) Hash Module: 3 -> Parallel Hash Join Hash Cond: (bar.b = foo.a) -> Parallel Seq Scan on bar -> Parallel Hash -> Broadcast Workers Motion 6:6 (slice3; segments: 6) -> Parallel Seq Scan on foo Optimizer: Postgres query optimizer (13 rows) For DEDUP_SEMI or DEDUP_SEMI_REVERSE join, each process need a unique RowIdExpr to identify unique rows, which is assigned with a baseline when building paths. It's ok for non-parallel plan, but in parallel mode there are multiple processes on same segment, RowIdExpr in not unique then. To enable that, add ParallelWorkerNumberOfSlice to identify worker id of a parallel plan of a slice. When rowidexpr is used, it's executed by 48 bits and left other 16 bits for segment_id. In parallel mode, we have to make more room for parallel worker id within segment_id's bits. This is done during planner with checks, in case that there are many segments with many parallel workers (which rarely happens). Authored-by: Zhang Mingli [email protected]
Authored-by: Zhang Mingli [email protected]
511df86
to
c348b2c
Compare
ENV problem, corrected. |
To handle semi join in parallel, we have enabled Parallel Semi join.
This commit introduces a new approach to process Semi Join Query in Parallel of MPP, enable DEDUP_SEMI and DEDUP_SEMI_REVERSE join in parallel to handle semi join cases which could be a win in MPP mode.
Allow to use RowIdExpr in paths to process SEMI join query in parallel, and we could use Broadcast Motion in parallel too.
If one of that tables is relatively smaller than another, that will be a big win.
Even Redistribute both side(in parallel-oblivious plan) and unique it in upper plans, it may be a win compared to a SemiJoin Nodes.
Performance
Test of case[0] three times with 10 parallel number.
Plan
Both parallel-oblivious and parallel-aware are enabled, for examples of parallel-aware case:
DEDUP_SEMI
DEDUP_SEMI_REVERSE
For DEDUP_SEMI or DEDUP_SEMI_REVERSE join, each process need a unique RowIdExpr to identify unique rows, which is assigned with a baseline when building paths.
It's ok for non-parallel plan, but in parallel mode there are multiple processes on same segment, RowIdExpr in not unique then. To enable that, add ParallelWorkerNumberOfSlice to identify worker id of a parallel plan of a slice.
When rowidexpr is used, it's executed by 48 bits and left other 16 bits for segment_id. In parallel mode, we have to make more room for parallel worker id within segment_id's bits. This is done during planner with checks, in case that there are many segments with many parallel workers (which rarely happens).
[0] DDL & DML
Authored-by: Zhang Mingli [email protected]
fix #ISSUE_Number
Change logs
Describe your change clearly, including what problem is being solved or what feature is being added.
If it has some breaking backward or forward compatibility, please clary.
Why are the changes needed?
Describe why the changes are necessary.
Does this PR introduce any user-facing change?
If yes, please clarify the previous behavior and the change this PR proposes.
How was this patch tested?
Please detail how the changes were tested, including manual tests and any relevant unit or integration tests.
Contributor's Checklist
Here are some reminders and checklists before/when submitting your pull request, please check them:
make installcheck
make -C src/test installcheck-cbdb-parallel
cloudberrydb/dev
team for review and approval when your PR is ready🥳