[feat](nereids) push down encode slot #44748

englefly · 2024-11-28T11:39:57Z

What problem does this PR solve?

in previous pr (#42408), string type of group_by and order_by keys are wrapped by encode_as_int(A) function.
In order to achieve better performance, it is better to push encode_as_int() down, because BE processes integers more efficiently.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

englefly · 2024-11-28T11:40:06Z

run buildall

doris-robot · 2024-11-28T11:40:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

morrySnow

should add test cases to enumerate child's plan type. such as union, except, intersect, window, generate, etc..

fe/fe-core/src/main/java/org/apache/doris/nereids/properties/LogicalProperties.java

...ore/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/EncodeStr.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/Plan.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalOlapScan.java

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalProject.java

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownEncodeSlot.java

morrySnow · 2024-11-29T04:36:18Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushDownEncodeSlot.java

+            return plan;
+        }
+
+        private Set<Slot> getByPassSlot(Plan plan) {


this function's name is diff with its action. it remove all bypass slot indeed.

englefly · 2024-12-04T09:19:23Z

run buildall

englefly · 2024-12-04T10:33:49Z

run buildall

englefly · 2024-12-04T10:58:23Z

run buildall

englefly · 2024-12-04T11:08:29Z

run buildall

englefly · 2024-12-08T15:02:07Z

run buildall

englefly · 2024-12-09T06:32:15Z

run buildall

englefly · 2024-12-09T11:02:58Z

run buildall

englefly · 2024-12-10T07:12:58Z

run buildall

englefly · 2024-12-10T14:37:56Z

run buildall

englefly · 2024-12-12T01:53:33Z

run buildall

englefly · 2024-12-12T06:05:53Z

run cloud_p0

englefly · 2024-12-18T07:40:49Z

run buildall

github-actions · 2024-12-18T07:41:07Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

github-actions · 2024-12-18T09:43:42Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

englefly · 2024-12-18T09:44:35Z

run buildall

github-actions · 2024-12-18T09:45:04Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

englefly · 2024-12-19T05:18:51Z

run buildall

github-actions · 2024-12-19T05:19:19Z

`sh-checker report`

To get the full details, please check in the job output.

shellcheck errors


'shellcheck ' returned error 1 finding the following syntactical issues:

----------

In gensrc/script/gen_build_version.sh line 38:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.


In gensrc/script/gen_build_version.sh line 228:
if [[ ${build_version_hotfix} > 0 ]]; then
                              ^-- SC2071 (error): > is for string comparisons. Use -gt instead.

For more information:
  https://www.shellcheck.net/wiki/SC2071 -- > is for string comparisons. Use ...
----------

You can address the above issues in one of three ways:
1. Manually correct the issue in the offending shell script;
2. Disable specific issues by adding the comment:
  # shellcheck disable=NNNN
above the line that contains the issue, where NNNN is the error code;
3. Add '-e NNNN' to the SHELLCHECK_OPTS setting in your .yml action file.

shfmt errors

'shfmt ' found no issues.

…oovy

englefly · 2024-12-19T14:41:49Z

run buildall

englefly · 2024-12-20T05:35:52Z

run buildall

englefly · 2024-12-22T02:58:17Z

run buildall

doris-robot · 2024-12-22T03:37:48Z

TPC-H: Total hot run time: 39537 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3abcc39b807b930bd3937466779318ce0fcd5773, data reload: false

------ Round 1 ----------------------------------
q1	17593	7166	7065	7065
q2	2043	172	162	162
q3	10573	1095	1145	1095
q4	10448	718	787	718
q5	7616	2682	2633	2633
q6	238	148	145	145
q7	962	621	613	613
q8	9266	1847	1916	1847
q9	6708	6442	6386	6386
q10	6986	2297	2291	2291
q11	465	261	248	248
q12	424	226	231	226
q13	17799	2941	2998	2941
q14	261	220	208	208
q15	551	494	495	494
q16	652	610	621	610
q17	981	532	475	475
q18	7447	7157	6866	6866
q19	1364	972	909	909
q20	468	186	184	184
q21	4067	3163	3112	3112
q22	376	309	314	309
Total cold run time: 107288 ms
Total hot run time: 39537 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7021	6983	7030	6983
q2	326	235	231	231
q3	2934	2793	2896	2793
q4	2046	1811	1815	1811
q5	5683	5674	5638	5638
q6	229	147	152	147
q7	2222	1812	1797	1797
q8	3337	3529	3499	3499
q9	8950	9020	8945	8945
q10	3598	3509	3566	3509
q11	601	496	535	496
q12	832	637	617	617
q13	11529	3144	3091	3091
q14	316	279	297	279
q15	565	529	512	512
q16	713	656	641	641
q17	1826	1635	1626	1626
q18	8267	7810	7749	7749
q19	1722	1562	1650	1562
q20	2045	1866	1898	1866
q21	5566	5415	5489	5415
q22	631	573	585	573
Total cold run time: 70959 ms
Total hot run time: 59780 ms

englefly · 2024-12-22T07:23:54Z

run p0

englefly · 2024-12-22T07:24:27Z

run cloud_p0

englefly · 2024-12-22T07:40:31Z

run buildall

doris-robot · 2024-12-22T08:20:05Z

TPC-H: Total hot run time: 39800 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5de42105385f3838b790dd54a5c9cdab07ab8ef0, data reload: false

------ Round 1 ----------------------------------
q1	17657	7276	7128	7128
q2	2046	170	173	170
q3	10573	1121	1172	1121
q4	10589	824	749	749
q5	7638	2691	2724	2691
q6	242	150	149	149
q7	971	636	612	612
q8	9249	1884	1925	1884
q9	6703	6479	6415	6415
q10	7050	2297	2290	2290
q11	472	266	257	257
q12	423	229	238	229
q13	17802	2911	3103	2911
q14	236	218	224	218
q15	550	507	499	499
q16	681	599	577	577
q17	981	579	567	567
q18	7198	6687	6606	6606
q19	1360	1110	953	953
q20	456	188	177	177
q21	4043	3419	3283	3283
q22	385	315	314	314
Total cold run time: 107305 ms
Total hot run time: 39800 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7078	7050	7060	7050
q2	336	230	234	230
q3	2909	2768	2913	2768
q4	2050	1818	1794	1794
q5	5660	5675	5636	5636
q6	226	150	145	145
q7	2282	1832	1821	1821
q8	3394	3568	3494	3494
q9	8994	9072	9020	9020
q10	3599	3898	3544	3544
q11	601	495	515	495
q12	848	593	611	593
q13	13385	3076	3078	3076
q14	307	273	277	273
q15	563	513	498	498
q16	683	644	673	644
q17	1866	1617	1626	1617
q18	8396	7897	7714	7714
q19	1728	1531	1610	1531
q20	2029	1935	1911	1911
q21	5597	5588	5505	5505
q22	663	605	592	592
Total cold run time: 73194 ms
Total hot run time: 59951 ms

englefly · 2024-12-22T12:41:49Z

run p0

englefly · 2024-12-22T12:58:59Z

run buildall

doris-robot · 2024-12-22T13:38:41Z

TPC-H: Total hot run time: 40064 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 784c841596082857ab8887c45c3e0f5312cf6842, data reload: false

------ Round 1 ----------------------------------
q1	17630	7389	7125	7125
q2	2042	169	166	166
q3	10855	1056	1174	1056
q4	10431	745	720	720
q5	7611	2675	2725	2675
q6	240	144	157	144
q7	999	624	603	603
q8	9242	1860	1938	1860
q9	6663	6379	6413	6379
q10	7028	2312	2342	2312
q11	473	267	260	260
q12	447	233	233	233
q13	17773	2955	2947	2947
q14	256	213	210	210
q15	563	499	497	497
q16	672	595	580	580
q17	963	543	521	521
q18	7289	6818	6927	6818
q19	1343	1122	1065	1065
q20	461	196	192	192
q21	4146	3411	3383	3383
q22	389	322	318	318
Total cold run time: 107516 ms
Total hot run time: 40064 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7011	7040	7025	7025
q2	330	236	236	236
q3	2919	2770	2962	2770
q4	2027	1777	1755	1755
q5	5699	5749	5647	5647
q6	227	149	149	149
q7	2279	1808	1826	1808
q8	3370	3591	3518	3518
q9	8934	9004	9002	9002
q10	3614	3543	3539	3539
q11	600	505	513	505
q12	803	673	616	616
q13	13127	3155	3115	3115
q14	335	272	291	272
q15	566	512	511	511
q16	710	658	655	655
q17	1859	1632	1613	1613
q18	8393	7808	7647	7647
q19	2089	1645	1656	1645
q20	2127	1865	1905	1865
q21	5563	5447	5448	5447
q22	666	569	591	569
Total cold run time: 73248 ms
Total hot run time: 59909 ms

englefly · 2024-12-23T03:47:33Z

run buildall

doris-robot · 2024-12-23T04:09:34Z

TPC-H: Total hot run time: 39435 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0faf7cf45351393588fe9adcc4663c1c5200c06, data reload: false

------ Round 1 ----------------------------------
q1	17638	7116	7064	7064
q2	2036	170	168	168
q3	10567	1129	1145	1129
q4	10228	755	704	704
q5	7624	2697	2667	2667
q6	236	143	142	142
q7	973	633	605	605
q8	9261	1823	1918	1823
q9	6673	6422	6418	6418
q10	6996	2270	2298	2270
q11	475	263	265	263
q12	415	220	232	220
q13	17783	2948	2974	2948
q14	240	210	212	210
q15	554	505	490	490
q16	674	580	599	580
q17	968	581	459	459
q18	7366	6677	6617	6617
q19	1414	997	1014	997
q20	454	185	189	185
q21	4047	3163	3216	3163
q22	372	313	318	313
Total cold run time: 106994 ms
Total hot run time: 39435 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7000	6980	6989	6980
q2	337	227	233	227
q3	2898	2691	2779	2691
q4	2038	1959	1868	1868
q5	5673	5630	5620	5620
q6	223	135	137	135
q7	2246	1789	1790	1789
q8	3331	3495	3525	3495
q9	8909	8929	8836	8836
q10	3625	3535	3485	3485
q11	597	550	490	490
q12	794	609	632	609
q13	11871	3054	3145	3054
q14	315	294	269	269
q15	560	518	505	505
q16	705	653	624	624
q17	1819	1598	1598	1598
q18	8122	7422	7408	7408
q19	1692	1505	1506	1505
q20	2073	1826	1856	1826
q21	5413	5297	5284	5284
q22	631	571	618	571
Total cold run time: 70872 ms
Total hot run time: 58869 ms

doris-robot · 2024-12-23T04:21:59Z

TPC-DS: Total hot run time: 195468 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0faf7cf45351393588fe9adcc4663c1c5200c06, data reload: false

query1	985	381	384	381
query2	6520	2404	2332	2332
query3	6715	235	215	215
query4	37076	26804	26776	26776
query5	4403	461	443	443
query6	276	196	200	196
query7	4628	321	308	308
query8	308	254	237	237
query9	9645	2697	2703	2697
query10	484	263	254	254
query11	20380	17434	17343	17343
query12	166	109	107	107
query13	1704	431	434	431
query14	11309	7636	6756	6756
query15	295	184	189	184
query16	8305	443	442	442
query17	1727	536	516	516
query18	1886	301	294	294
query19	373	162	152	152
query20	120	111	109	109
query21	203	103	108	103
query22	4386	4270	4143	4143
query23	34495	33615	33479	33479
query24	11707	2347	2155	2155
query25	652	401	405	401
query26	1658	156	154	154
query27	2664	325	327	325
query28	7594	2415	2411	2411
query29	1001	406	416	406
query30	369	180	180	180
query31	1049	813	776	776
query32	98	62	61	61
query33	780	293	292	292
query34	960	506	513	506
query35	839	722	718	718
query36	1096	953	948	948
query37	262	79	82	79
query38	4147	4085	4221	4085
query39	1493	1443	1427	1427
query40	289	102	104	102
query41	49	47	49	47
query42	120	103	107	103
query43	548	509	486	486
query44	1308	828	810	810
query45	187	166	179	166
query46	1199	747	708	708
query47	1976	1820	1852	1820
query48	428	318	340	318
query49	1276	385	388	385
query50	1217	656	689	656
query51	7240	7129	7077	7077
query52	103	95	88	88
query53	253	181	187	181
query54	1317	418	415	415
query55	79	80	82	80
query56	267	241	243	241
query57	1263	1132	1148	1132
query58	236	209	221	209
query59	3306	3060	3155	3060
query60	278	252	244	244
query61	112	105	109	105
query62	901	696	697	696
query63	219	186	191	186
query64	5011	684	668	668
query65	3258	3165	3178	3165
query66	1365	330	323	323
query67	15856	15542	15435	15435
query68	5415	559	545	545
query69	437	315	258	258
query70	1386	1306	1216	1216
query71	443	250	254	250
query72	6431	4036	4197	4036
query73	782	366	367	366
query74	10294	8806	8805	8805
query75	3461	2649	2636	2636
query76	3387	1245	1153	1153
query77	593	293	290	290
query78	10199	9376	9483	9376
query79	2627	600	604	600
query80	1110	433	419	419
query81	556	246	234	234
query82	696	121	114	114
query83	233	148	144	144
query84	238	76	75	75
query85	1586	309	312	309
query86	453	287	310	287
query87	4589	4674	4292	4292
query88	4193	2198	2184	2184
query89	417	296	292	292
query90	2158	189	195	189
query91	142	107	104	104
query92	63	55	53	53
query93	1749	544	544	544
query94	920	292	281	281
query95	350	249	246	246
query96	625	283	284	283
query97	2880	2724	2643	2643
query98	215	202	235	202
query99	1553	1314	1329	1314
Total cold run time: 312600 ms
Total hot run time: 195468 ms

doris-robot · 2024-12-23T04:27:29Z

ClickBench: Total hot run time: 32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0faf7cf45351393588fe9adcc4663c1c5200c06, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.08	0.06
query4	1.60	0.11	0.10
query5	0.44	0.40	0.42
query6	1.14	0.68	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.49
query10	0.56	0.56	0.57
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.60
query14	2.73	2.82	2.83
query15	0.89	0.82	0.84
query16	0.38	0.37	0.38
query17	1.02	1.01	1.04
query18	0.23	0.21	0.21
query19	1.93	1.88	2.08
query20	0.02	0.01	0.01
query21	15.36	0.58	0.56
query22	2.91	2.34	1.83
query23	16.81	1.00	0.81
query24	3.53	1.45	0.30
query25	0.19	0.14	0.13
query26	0.44	0.15	0.13
query27	0.04	0.05	0.04
query28	10.81	1.10	1.07
query29	12.71	3.23	3.23
query30	0.25	0.06	0.07
query31	2.87	0.38	0.38
query32	3.24	0.46	0.46
query33	3.10	3.04	3.06
query34	16.84	4.42	4.46
query35	4.46	4.47	4.48
query36	0.69	0.47	0.50
query37	0.09	0.06	0.06
query38	0.05	0.03	0.04
query39	0.04	0.03	0.03
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.04
Total cold run time: 107.56 s
Total hot run time: 32 s

morrySnow reviewed Nov 29, 2024

View reviewed changes

englefly force-pushed the pushencode branch 3 times, most recently from 00508e8 to 06337a7 Compare December 3, 2024 09:45

englefly force-pushed the pushencode branch from c786601 to 7efb5c5 Compare December 4, 2024 10:58

englefly force-pushed the pushencode branch from 7efb5c5 to 4be5093 Compare December 4, 2024 11:08

englefly force-pushed the pushencode branch 2 times, most recently from 2842018 to 51b0083 Compare December 11, 2024 12:46

englefly force-pushed the pushencode branch from b3f9f84 to 64593c7 Compare December 18, 2024 07:40

englefly added 6 commits December 19, 2024 21:01

todo regression-test/suites/correctness_p0/test_first_value_window.gr…

04b8e87

…oovy

if shape check failed, print memo

a9c1fff

print all plan

bdd7ea8

try merge project with simplify encode decode

a2031c7

add log

dadb867

force proj->topn->agg

321f4f2

englefly force-pushed the pushencode branch from df14770 to 321f4f2 Compare December 19, 2024 13:02

shape2-proj-topn-agg

d472eae

englefly added 2 commits December 20, 2024 13:30

fix coalease(window())

a784ed0

fmt

6b06199

fix proj-proj

3abcc39

test ds49

5de4210

add log

784c841

print all plan when shape check failed

b0faf7c

englefly force-pushed the pushencode branch from 22d4eec to b0faf7c Compare December 23, 2024 03:47

[feat](nereids) push down encode slot #44748

Are you sure you want to change the base?

[feat](nereids) push down encode slot #44748

Conversation

englefly commented Nov 28, 2024

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

englefly commented Nov 28, 2024

doris-robot commented Nov 28, 2024

morrySnow left a comment

Choose a reason for hiding this comment

morrySnow Nov 29, 2024

Choose a reason for hiding this comment

englefly commented Dec 4, 2024

englefly commented Dec 4, 2024

englefly commented Dec 4, 2024

englefly commented Dec 4, 2024

englefly commented Dec 8, 2024

englefly commented Dec 9, 2024

englefly commented Dec 9, 2024

englefly commented Dec 10, 2024

englefly commented Dec 10, 2024

englefly commented Dec 12, 2024

englefly commented Dec 12, 2024

englefly commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

github-actions bot commented Dec 18, 2024

sh-checker report

englefly commented Dec 18, 2024

github-actions bot commented Dec 18, 2024

sh-checker report

englefly commented Dec 19, 2024

github-actions bot commented Dec 19, 2024

sh-checker report

englefly commented Dec 19, 2024

englefly commented Dec 20, 2024

englefly commented Dec 22, 2024

doris-robot commented Dec 22, 2024

englefly commented Dec 22, 2024

englefly commented Dec 22, 2024

englefly commented Dec 22, 2024

doris-robot commented Dec 22, 2024

englefly commented Dec 22, 2024

englefly commented Dec 22, 2024

doris-robot commented Dec 22, 2024

englefly commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

doris-robot commented Dec 23, 2024

`sh-checker report`

`sh-checker report`

`sh-checker report`

`sh-checker report`