Doris support deducing partitions #280
Conversation
|
@BlockLiu please help me code review, thx! |
| .collect(Collectors.toList()) | ||
| ); | ||
| DorisPartitionTemplate dorisPartitionTemplate = writerConfiguration.getNecessaryOption(DorisWriterOptions.PARTITIONS, CommonErrorCode.CONFIG_ERROR); | ||
| List<DorisPartition> dorisPartitions = parseTemplateToDorisPartitions(dorisPartitionTemplate); |
There was a problem hiding this comment.
Great work, but I have one question.
- Old partition option support flexible partition range. For example, ["start_range": "2022-01-01", "end_range": "2022-02-01"].
- In this PR,
DorisSinkalways generate partitions of single day. - So could you add a switch option to decide if using this template?
There was a problem hiding this comment.
@BlockLiu Thanks for your suggestion, now I understand what you mean.
So if I add a parameter representing the day, week, month, quarter and year to the template to represent the step size of the partition range , is it okay? For example, if use "month" , DorisSink will generate partitions by one month.
I think there are very few scenes to partition by random number of days like 4 days or 11 days.
There was a problem hiding this comment.
Actually, I mean we can keep both two method for setting doris partition(s).
- The first way is deducing partition as your codes.
- And the second way is the original method, _i.e., directly setting doris partition.
So we don't need a "step size", but an option to decide which way to use.
There was a problem hiding this comment.
Like this:
bool useDeducedPartition = xxx // get from job configuration
if (useDeducedPartition) {
// use your codes
} else {
// use the original codes
}
| @@ -0,0 +1,25 @@ | |||
| package com.bytedance.bitsail.connector.doris.partition; | |||
There was a problem hiding this comment.
There should be a license header. You can copy from other source files.
|
It seems that it has not been updated for a while, and I look forward to your continuous contribution. @beyond-up |
beyond-up
force-pushed
the
dev_doris_partition
branch
from
fc240c6 to
20e6007
Compare
# Conflicts: # bitsail-connectors/connector-doris/src/main/java/com/bytedance/bitsail/connector/doris/option/DorisWriterOptions.java # bitsail-connectors/connector-doris/src/main/java/com/bytedance/bitsail/connector/doris/sink/DorisSink.java
#149
Signed-off-by:
Pre-Checklist
Note: Please complete ALL items in the following checklist.
Purpose
Some description about what this PR wants to do.
Approaches
Some description about how this PR achives the purpose.
Related Issues
#149
New Behavior (screenshots if needed)
N/A