HIVE-29166: Fix the partition column update logic in ConvertJoinMapJoin#convertJoinBucketMapJoin. #6048
+222
−20
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
okumin
reviewed
Sep 1, 2025
| key expressions: _col1 (type: string), _col2 (type: string) | ||
| null sort order: zz | ||
| sort order: ++ | ||
| Map-reduce partition columns: _col2 (type: string), _col1 (type: string) |
There was a problem hiding this comment.
I verified that this list is inverted on the master branch, and the result would be empty.
Map-reduce partition columns: _col1 (type: string), _col2 (type: string)
| BucketMapJoin:true,Conds:SEL_51._col1, _col2=RS_49._col1, _col2(Inner),Output:["_col0","_col1","_col2","_col3","_col4","_col5"] | ||
| <-Map 2 [CUSTOM_EDGE] vectorized | ||
| MULTICAST [RS_49] | ||
| PartitionCols:_col2, _col1 |
There was a problem hiding this comment.
I verified that this list is inverted on the master branch, and the result would be empty.
PartitionCols:_col1, _col2
kasakrisz
approved these changes
Sep 4, 2025
|
Thank you for the review. If it is OK, can we merge it to master? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Modify the partition column update logic in
ConvertJoinMapJoin#convertJoinBucketMapJoinso that small table's partitionCols are updated if they do not match the big table's bucketCols.Previously, the partition columns were updated only when the number of partition columns is not equal to the number of bucket columns. However, this could cause incorrect bucket routing when the counts are equal but the column orders differ. For example, if the bucket columns are {id, part} and the partition columns are {part, id}, then the partition columns should be updated to {id, part}, but the current implementation does not update them.
The new logic now updates the partition columns if the counts are not equal or the column orders differ.
Why are the changes needed?
As described above, the current BucketMapJoin conversion logic may not update the partition columns properly, which leads to incorrect results, as reported in the JIRA.
Does this PR introduce any user-facing change?
No
How was this patch tested?
I added two qfiles in which the current Hive returns incorrect results. I also checked that the patch fixes the originally reported duplicate-generation issue.