Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](correctness) Fix operator initialization #45728

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Dec 20, 2024

What problem does this PR solve?

Now we plan for local exchange depends on operator initialization. This PR fixed a wrong order between those two steps.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39940 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 91c862b6ac109403946a7feb08e0230f1e64b078, data reload: false

------ Round 1 ----------------------------------
q1	17677	7540	7332	7332
q2	2062	173	161	161
q3	10630	1147	1168	1147
q4	10237	735	678	678
q5	7629	2705	2783	2705
q6	241	154	148	148
q7	988	621	609	609
q8	9241	1861	1924	1861
q9	6673	6385	6431	6385
q10	7006	2302	2314	2302
q11	475	253	259	253
q12	432	217	216	216
q13	17783	2966	2970	2966
q14	257	209	207	207
q15	558	498	486	486
q16	659	605	600	600
q17	1011	545	524	524
q18	7531	6732	6800	6732
q19	1387	1017	1047	1017
q20	488	179	180	179
q21	4318	3351	3122	3122
q22	393	310	328	310
Total cold run time: 107676 ms
Total hot run time: 39940 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7353	7256	7241	7241
q2	334	230	231	230
q3	2940	2821	2757	2757
q4	1962	1759	1721	1721
q5	5367	5366	5443	5366
q6	227	144	144	144
q7	2128	1680	1727	1680
q8	3240	3380	3432	3380
q9	8637	8613	8579	8579
q10	3514	3417	3444	3417
q11	603	498	502	498
q12	787	577	555	555
q13	11104	2923	2972	2923
q14	291	260	261	260
q15	550	503	494	494
q16	673	652	641	641
q17	1813	1592	1574	1574
q18	7977	7381	7576	7381
q19	1689	1615	1450	1450
q20	2060	1826	1817	1817
q21	5402	5433	5278	5278
q22	660	594	580	580
Total cold run time: 69311 ms
Total hot run time: 57966 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.88% (10133/26062)
Line Coverage: 29.80% (85265/286097)
Region Coverage: 28.93% (43544/150491)
Branch Coverage: 25.45% (22176/87148)
Coverage Report: http://coverage.selectdb-in.cc/coverage/91c862b6ac109403946a7feb08e0230f1e64b078_91c862b6ac109403946a7feb08e0230f1e64b078/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 190248 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 91c862b6ac109403946a7feb08e0230f1e64b078, data reload: false

query1	1003	403	381	381
query2	6520	2504	2374	2374
query3	6710	220	211	211
query4	33921	23518	23486	23486
query5	4340	454	469	454
query6	286	193	189	189
query7	4658	307	308	307
query8	303	241	231	231
query9	9606	2724	2709	2709
query10	488	253	249	249
query11	18028	15282	15119	15119
query12	177	104	107	104
query13	1669	438	433	433
query14	9808	7306	7604	7306
query15	306	178	189	178
query16	8308	462	489	462
query17	1796	595	569	569
query18	2154	297	336	297
query19	355	149	148	148
query20	116	106	110	106
query21	212	103	101	101
query22	4486	4365	4375	4365
query23	34306	33489	33217	33217
query24	11297	2461	2446	2446
query25	667	386	372	372
query26	1778	152	152	152
query27	2879	321	332	321
query28	7793	2401	2436	2401
query29	1018	408	398	398
query30	328	146	149	146
query31	1037	811	820	811
query32	94	58	55	55
query33	770	288	281	281
query34	964	509	536	509
query35	946	729	737	729
query36	1101	949	960	949
query37	204	75	76	75
query38	4246	4085	4206	4085
query39	1497	1435	1427	1427
query40	283	99	97	97
query41	47	46	46	46
query42	115	104	102	102
query43	517	501	479	479
query44	1284	789	799	789
query45	181	168	162	162
query46	1158	695	710	695
query47	1945	1854	1842	1842
query48	411	328	329	328
query49	1275	375	395	375
query50	811	375	388	375
query51	7035	7119	7106	7106
query52	104	94	92	92
query53	253	178	185	178
query54	1225	408	408	408
query55	85	77	77	77
query56	253	242	232	232
query57	1245	1108	1112	1108
query58	234	210	228	210
query59	3314	2962	2990	2962
query60	279	247	231	231
query61	114	110	102	102
query62	891	703	660	660
query63	214	183	188	183
query64	5148	664	652	652
query65	3218	3219	3234	3219
query66	1389	305	309	305
query67	15858	15540	15511	15511
query68	5183	545	538	538
query69	430	253	312	253
query70	1151	1119	1128	1119
query71	444	243	258	243
query72	6855	4032	4170	4032
query73	801	361	360	360
query74	10431	8783	8847	8783
query75	3476	2622	2650	2622
query76	3164	1063	1115	1063
query77	558	311	278	278
query78	10054	9449	9339	9339
query79	1828	585	607	585
query80	1068	426	434	426
query81	533	228	233	228
query82	965	114	123	114
query83	241	149	140	140
query84	234	72	79	72
query85	1302	306	293	293
query86	433	282	293	282
query87	4511	4511	4470	4470
query88	3508	2251	2191	2191
query89	408	293	305	293
query90	2156	190	187	187
query91	133	106	117	106
query92	110	51	53	51
query93	1757	537	537	537
query94	1037	285	248	248
query95	363	249	252	249
query96	614	273	291	273
query97	2899	2671	2669	2669
query98	220	198	196	196
query99	1555	1329	1329	1329
Total cold run time: 303483 ms
Total hot run time: 190248 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 91c862b6ac109403946a7feb08e0230f1e64b078, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.08	0.07
query4	1.59	0.10	0.10
query5	0.44	0.43	0.41
query6	1.15	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.48
query10	0.58	0.57	0.55
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.62	0.60	0.61
query14	2.90	2.70	2.74
query15	0.89	0.82	0.83
query16	0.37	0.38	0.39
query17	1.07	1.02	1.04
query18	0.23	0.20	0.20
query19	1.90	1.75	1.99
query20	0.01	0.01	0.02
query21	15.37	0.58	0.57
query22	2.71	2.62	1.67
query23	16.82	1.04	0.85
query24	3.43	1.48	1.81
query25	0.27	0.12	0.16
query26	0.52	0.13	0.14
query27	0.05	0.05	0.04
query28	9.54	1.10	1.08
query29	12.53	3.29	3.22
query30	0.25	0.06	0.06
query31	2.87	0.38	0.38
query32	3.24	0.47	0.45
query33	3.07	3.03	3.11
query34	17.00	4.50	4.57
query35	4.48	4.50	4.52
query36	0.68	0.47	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.17	0.13	0.14
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 106.3 s
Total hot run time: 32.91 s

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39473 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01700c713b9d73df34996d20ed76db564cbac4ac, data reload: false

------ Round 1 ----------------------------------
q1	17574	7488	7268	7268
q2	2046	181	188	181
q3	10529	1110	1095	1095
q4	10216	731	717	717
q5	7582	2689	2728	2689
q6	236	152	154	152
q7	991	634	604	604
q8	9237	1831	1940	1831
q9	6706	6455	6401	6401
q10	6930	2284	2279	2279
q11	464	256	256	256
q12	442	225	224	224
q13	17786	2866	2986	2866
q14	236	204	215	204
q15	561	510	503	503
q16	653	603	576	576
q17	990	653	490	490
q18	7251	6671	6704	6671
q19	1350	1044	933	933
q20	489	191	190	190
q21	4078	3199	3034	3034
q22	382	309	315	309
Total cold run time: 106729 ms
Total hot run time: 39473 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7268	7245	7235	7235
q2	331	231	230	230
q3	2924	2752	2892	2752
q4	2162	1833	1856	1833
q5	5546	5712	5631	5631
q6	227	145	143	143
q7	2194	1754	1827	1754
q8	3352	3571	3542	3542
q9	8951	8974	8859	8859
q10	3643	3539	3565	3539
q11	603	514	515	514
q12	791	653	637	637
q13	14195	3128	3177	3128
q14	295	270	261	261
q15	553	501	488	488
q16	687	652	636	636
q17	1872	1640	1582	1582
q18	7915	7522	7591	7522
q19	1683	1517	1471	1471
q20	2059	1820	1823	1820
q21	5534	5292	5305	5292
q22	656	585	581	581
Total cold run time: 73441 ms
Total hot run time: 59450 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.87% (10131/26063)
Line Coverage: 29.80% (85278/286122)
Region Coverage: 28.93% (43544/150501)
Branch Coverage: 25.44% (22168/87154)
Coverage Report: http://coverage.selectdb-in.cc/coverage/01700c713b9d73df34996d20ed76db564cbac4ac_01700c713b9d73df34996d20ed76db564cbac4ac/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 189745 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01700c713b9d73df34996d20ed76db564cbac4ac, data reload: false

query1	981	386	374	374
query2	6533	2496	2400	2400
query3	6701	232	225	225
query4	33950	23539	23250	23250
query5	4352	464	446	446
query6	273	203	195	195
query7	4624	305	321	305
query8	318	243	230	230
query9	9632	2707	2697	2697
query10	494	260	278	260
query11	18051	15100	15035	15035
query12	159	107	103	103
query13	1673	421	435	421
query14	9141	7113	7748	7113
query15	310	173	202	173
query16	8399	458	475	458
query17	1796	586	571	571
query18	2162	330	297	297
query19	363	157	146	146
query20	119	109	110	109
query21	210	104	106	104
query22	4696	4407	4519	4407
query23	34445	33148	33100	33100
query24	11318	2465	2494	2465
query25	685	375	395	375
query26	1874	154	154	154
query27	2915	325	314	314
query28	7916	2411	2400	2400
query29	1051	403	399	399
query30	313	147	153	147
query31	1033	810	811	810
query32	98	55	82	55
query33	783	295	308	295
query34	974	515	522	515
query35	877	743	751	743
query36	1084	958	934	934
query37	295	79	79	79
query38	4277	4244	4322	4244
query39	1469	1439	1428	1428
query40	280	99	97	97
query41	47	44	47	44
query42	116	103	103	103
query43	533	492	485	485
query44	1253	836	808	808
query45	189	168	166	166
query46	1151	718	686	686
query47	1923	1824	1859	1824
query48	406	320	313	313
query49	1291	404	390	390
query50	791	384	373	373
query51	7214	7156	7114	7114
query52	103	93	89	89
query53	261	183	183	183
query54	1262	402	412	402
query55	84	81	80	80
query56	266	244	248	244
query57	1267	1106	1095	1095
query58	237	214	225	214
query59	3469	3086	3000	3000
query60	270	254	284	254
query61	108	104	106	104
query62	902	673	675	673
query63	215	185	184	184
query64	4982	668	634	634
query65	3226	3183	3239	3183
query66	1452	310	308	308
query67	15838	15554	15568	15554
query68	5840	555	540	540
query69	421	252	247	247
query70	1206	1010	1038	1010
query71	461	246	277	246
query72	6427	4067	4093	4067
query73	785	350	360	350
query74	10334	8818	8821	8818
query75	3454	2624	2631	2624
query76	3557	1076	1033	1033
query77	559	263	306	263
query78	10273	9509	9355	9355
query79	1866	604	614	604
query80	1038	427	441	427
query81	537	245	235	235
query82	950	115	118	115
query83	247	147	148	147
query84	234	68	69	68
query85	1414	311	305	305
query86	382	299	294	294
query87	4568	4332	4373	4332
query88	3546	2191	2164	2164
query89	395	290	294	290
query90	2067	198	183	183
query91	136	105	115	105
query92	62	52	52	52
query93	1245	543	529	529
query94	1085	285	275	275
query95	361	243	255	243
query96	614	286	274	274
query97	2869	2678	2665	2665
query98	224	198	199	198
query99	1584	1316	1307	1307
Total cold run time: 304221 ms
Total hot run time: 189745 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.32 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01700c713b9d73df34996d20ed76db564cbac4ac, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.03
query3	0.23	0.07	0.07
query4	1.60	0.11	0.10
query5	0.41	0.41	0.41
query6	1.17	0.64	0.63
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.59	0.51	0.49
query10	0.56	0.58	0.56
query11	0.14	0.10	0.11
query12	0.14	0.11	0.11
query13	0.61	0.63	0.60
query14	2.84	2.76	2.85
query15	0.88	0.82	0.82
query16	0.38	0.38	0.37
query17	1.08	1.01	1.03
query18	0.22	0.21	0.20
query19	1.95	1.83	2.01
query20	0.01	0.01	0.01
query21	15.38	0.58	0.58
query22	2.96	2.35	2.42
query23	16.92	1.08	0.79
query24	3.19	1.11	2.08
query25	0.30	0.11	0.09
query26	0.56	0.13	0.12
query27	0.05	0.05	0.05
query28	9.63	1.11	1.07
query29	12.59	3.21	3.19
query30	0.25	0.06	0.05
query31	2.87	0.38	0.39
query32	3.25	0.47	0.46
query33	3.08	3.10	3.14
query34	17.06	4.51	4.52
query35	4.54	4.51	4.54
query36	0.65	0.48	0.50
query37	0.10	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.78 s
Total hot run time: 33.32 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants