Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[opt](profile) Disable profile for insert into values and other commands #47296

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

zhiqiang-hhhh
Copy link
Contributor

What problem does this PR solve?

Profile of INSERT INTO VALUES and other ddl commands is disabled.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32873 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4df958b58505de661b676a6bfc8855a6b8e37d59, data reload: false

------ Round 1 ----------------------------------
q1	17569	5608	5486	5486
q2	2044	325	172	172
q3	10402	1258	767	767
q4	10255	979	516	516
q5	8399	2405	2243	2243
q6	208	170	137	137
q7	933	771	613	613
q8	9249	1486	1279	1279
q9	5322	5012	4956	4956
q10	6872	2329	1887	1887
q11	479	278	255	255
q12	351	372	217	217
q13	17802	3794	3096	3096
q14	224	236	208	208
q15	530	481	471	471
q16	615	607	579	579
q17	570	904	319	319
q18	6915	6633	6669	6633
q19	3054	1025	584	584
q20	312	330	188	188
q21	2816	2205	1960	1960
q22	373	346	307	307
Total cold run time: 105294 ms
Total hot run time: 32873 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5855	5681	5733	5681
q2	253	328	232	232
q3	2283	2669	2288	2288
q4	1520	1842	1407	1407
q5	4343	4733	4890	4733
q6	185	169	129	129
q7	2141	1971	1829	1829
q8	2688	2949	2764	2764
q9	7267	7307	7282	7282
q10	3089	3214	2707	2707
q11	589	510	489	489
q12	687	827	674	674
q13	3603	3995	3290	3290
q14	295	305	286	286
q15	515	468	465	465
q16	651	678	643	643
q17	1264	1756	1292	1292
q18	7738	7484	7414	7414
q19	843	983	1246	983
q20	2019	2061	1937	1937
q21	5716	5423	5046	5046
q22	634	586	583	583
Total cold run time: 54178 ms
Total hot run time: 52154 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194441 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4df958b58505de661b676a6bfc8855a6b8e37d59, data reload: false

query1	1368	972	913	913
query2	6473	2092	2017	2017
query3	11094	4591	4605	4591
query4	32588	23194	22894	22894
query5	3476	603	486	486
query6	284	199	186	186
query7	3971	502	309	309
query8	298	241	235	235
query9	9062	2609	2604	2604
query10	459	309	248	248
query11	17776	15213	14905	14905
query12	152	106	106	106
query13	1576	526	399	399
query14	9414	7646	7252	7252
query15	239	239	193	193
query16	7868	687	467	467
query17	1617	798	548	548
query18	2098	415	307	307
query19	206	188	160	160
query20	128	126	112	112
query21	207	124	113	113
query22	4698	4651	4347	4347
query23	34684	34139	33340	33340
query24	6620	2266	2288	2266
query25	478	437	382	382
query26	728	251	159	159
query27	2047	502	343	343
query28	5435	2479	2480	2479
query29	604	535	417	417
query30	221	197	155	155
query31	955	876	811	811
query32	67	57	58	57
query33	481	365	312	312
query34	797	912	519	519
query35	853	854	794	794
query36	1014	1038	974	974
query37	130	103	78	78
query38	4358	4359	4303	4303
query39	1476	1458	1439	1439
query40	209	124	99	99
query41	50	47	49	47
query42	121	104	104	104
query43	519	526	503	503
query44	1379	819	829	819
query45	190	184	168	168
query46	889	1085	659	659
query47	1922	1951	1874	1874
query48	411	409	334	334
query49	719	492	431	431
query50	677	688	394	394
query51	6902	7145	7010	7010
query52	96	101	90	90
query53	219	254	217	217
query54	473	509	424	424
query55	83	81	79	79
query56	256	286	245	245
query57	1216	1203	1143	1143
query58	260	244	248	244
query59	3067	3231	2960	2960
query60	294	271	256	256
query61	136	141	119	119
query62	794	723	661	661
query63	218	183	181	181
query64	3224	1036	651	651
query65	3234	3160	3174	3160
query66	697	402	294	294
query67	15904	15740	15481	15481
query68	5234	875	507	507
query69	463	286	251	251
query70	1225	1144	1065	1065
query71	377	289	281	281
query72	5441	3921	3932	3921
query73	641	798	354	354
query74	10055	8977	9088	8977
query75	3244	3154	2661	2661
query76	3014	1234	790	790
query77	459	363	278	278
query78	10077	10049	9365	9365
query79	2783	814	605	605
query80	1394	614	444	444
query81	561	277	234	234
query82	362	150	121	121
query83	170	172	149	149
query84	234	83	73	73
query85	770	386	320	320
query86	459	323	305	305
query87	4351	4508	4451	4451
query88	4543	2166	2128	2128
query89	403	322	291	291
query90	1819	193	188	188
query91	133	143	106	106
query92	65	59	51	51
query93	2857	885	512	512
query94	709	468	296	296
query95	327	263	259	259
query96	491	659	292	292
query97	2793	2843	2820	2820
query98	230	202	209	202
query99	1308	1361	1277	1277
Total cold run time: 287151 ms
Total hot run time: 194441 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4df958b58505de661b676a6bfc8855a6b8e37d59, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.07	0.07
query4	1.61	0.10	0.10
query5	0.42	0.42	0.42
query6	1.16	0.64	0.65
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.60	0.50	0.49
query10	0.55	0.56	0.55
query11	0.15	0.10	0.10
query12	0.14	0.10	0.10
query13	0.61	0.60	0.60
query14	2.84	2.72	2.83
query15	0.90	0.81	0.82
query16	0.37	0.38	0.38
query17	1.07	1.03	1.05
query18	0.23	0.21	0.21
query19	1.90	1.80	1.90
query20	0.01	0.01	0.01
query21	15.35	0.92	0.56
query22	0.75	0.74	0.76
query23	15.24	1.43	0.55
query24	2.93	1.65	1.98
query25	0.17	0.12	0.15
query26	0.30	0.15	0.14
query27	0.08	0.05	0.05
query28	14.33	0.90	0.43
query29	12.55	3.90	3.24
query30	0.25	0.10	0.06
query31	2.80	0.61	0.38
query32	3.22	0.55	0.46
query33	2.95	2.98	3.02
query34	16.66	5.25	4.55
query35	4.55	4.61	4.56
query36	0.82	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.02	0.03
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.39 s
Total hot run time: 31.4 s

@zhiqiang-hhhh
Copy link
Contributor Author

run p0

yiguolei
yiguolei previously approved these changes Jan 22, 2025
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 22, 2025
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.


if (plan instanceof InsertIntoTableCommand) {
LogicalPlan logicalPlan = ((InsertIntoTableCommand) plan).getLogicalQuery();
if (logicalPlan instanceof UnboundTableSink) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if UnboundJdbcSink or UnboundHiveSink? is it safe for them to generate profile not like olap table?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed by adding UnboundBaseExternalTableSink

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jan 22, 2025
@doris-robot
Copy link

TPC-H: Total hot run time: 32048 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01dbb1c8e0ccbcd5b3371f77b8839d332287744e, data reload: false

------ Round 1 ----------------------------------
q1	17679	5564	5393	5393
q2	2053	298	168	168
q3	10411	1270	728	728
q4	10227	976	528	528
q5	7566	2366	2158	2158
q6	194	166	133	133
q7	918	755	594	594
q8	9256	1370	1125	1125
q9	5349	4996	4873	4873
q10	6827	2334	1890	1890
q11	479	279	257	257
q12	345	359	224	224
q13	17779	3677	3062	3062
q14	239	224	221	221
q15	523	477	453	453
q16	629	614	582	582
q17	560	872	309	309
q18	7094	6564	6337	6337
q19	1213	948	527	527
q20	316	336	201	201
q21	2794	2189	1972	1972
q22	361	334	313	313
Total cold run time: 102812 ms
Total hot run time: 32048 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5445	5500	5450	5450
q2	241	333	240	240
q3	2309	2677	2342	2342
q4	1400	1791	1355	1355
q5	4310	4791	4626	4626
q6	165	157	127	127
q7	2047	1993	1848	1848
q8	2679	2826	2691	2691
q9	7277	7248	7199	7199
q10	2989	3280	2781	2781
q11	593	527	486	486
q12	684	749	629	629
q13	3454	3963	3238	3238
q14	285	317	268	268
q15	515	479	459	459
q16	634	684	628	628
q17	1212	1733	1256	1256
q18	7775	7487	7349	7349
q19	779	1138	1066	1066
q20	2013	2015	1909	1909
q21	5724	5311	4907	4907
q22	632	596	590	590
Total cold run time: 53162 ms
Total hot run time: 51444 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187748 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01dbb1c8e0ccbcd5b3371f77b8839d332287744e, data reload: false

query1	977	389	380	380
query2	6527	2150	2117	2117
query3	6800	210	216	210
query4	33641	23274	22968	22968
query5	4323	604	474	474
query6	297	196	191	191
query7	4606	500	307	307
query8	297	246	236	236
query9	9360	2690	2667	2667
query10	480	315	259	259
query11	18213	15298	15048	15048
query12	149	105	106	105
query13	1659	519	411	411
query14	9549	7005	7216	7005
query15	258	196	187	187
query16	8103	628	452	452
query17	1623	745	571	571
query18	2125	390	289	289
query19	225	179	148	148
query20	115	109	110	109
query21	211	119	112	112
query22	4129	4179	4215	4179
query23	34045	33480	32865	32865
query24	6709	2225	2244	2225
query25	469	436	388	388
query26	1091	224	153	153
query27	2131	463	327	327
query28	5182	2462	2447	2447
query29	542	504	406	406
query30	227	187	169	169
query31	960	910	819	819
query32	88	63	59	59
query33	530	378	303	303
query34	735	846	509	509
query35	812	814	737	737
query36	997	1023	930	930
query37	126	102	83	83
query38	4250	4164	4049	4049
query39	1477	1425	1404	1404
query40	208	120	102	102
query41	60	58	50	50
query42	125	101	103	101
query43	522	528	506	506
query44	1298	797	804	797
query45	185	168	166	166
query46	850	1037	638	638
query47	1811	1816	1770	1770
query48	372	416	325	325
query49	780	487	406	406
query50	625	662	394	394
query51	6831	6799	6875	6799
query52	104	101	89	89
query53	225	248	190	190
query54	493	477	433	433
query55	80	79	83	79
query56	264	264	243	243
query57	1201	1161	1083	1083
query58	243	239	269	239
query59	3074	3166	3142	3142
query60	300	263	282	263
query61	149	115	115	115
query62	824	732	663	663
query63	223	199	187	187
query64	4553	1007	682	682
query65	3278	3142	3193	3142
query66	1068	416	299	299
query67	15697	15653	15407	15407
query68	4880	832	549	549
query69	470	290	263	263
query70	1198	1169	1138	1138
query71	380	278	269	269
query72	5874	3841	3567	3567
query73	645	750	364	364
query74	10297	9086	8868	8868
query75	3158	3190	2632	2632
query76	3162	1196	775	775
query77	515	360	268	268
query78	9887	10051	9353	9353
query79	2705	799	597	597
query80	592	562	443	443
query81	516	280	248	248
query82	450	155	125	125
query83	189	177	147	147
query84	243	95	82	82
query85	796	366	364	364
query86	401	326	304	304
query87	4550	4483	4356	4356
query88	4894	2196	2167	2167
query89	403	336	288	288
query90	1862	188	189	188
query91	141	138	170	138
query92	65	57	53	53
query93	1864	864	541	541
query94	684	397	281	281
query95	339	263	253	253
query96	489	609	283	283
query97	2731	2828	2767	2767
query98	226	203	210	203
query99	1261	1351	1266	1266
Total cold run time: 285348 ms
Total hot run time: 187748 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01dbb1c8e0ccbcd5b3371f77b8839d332287744e, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.61	0.10	0.11
query5	0.42	0.42	0.42
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.49
query10	0.55	0.57	0.57
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.59	0.60	0.59
query14	2.83	2.76	2.83
query15	0.89	0.81	0.82
query16	0.37	0.40	0.38
query17	0.96	1.03	1.02
query18	0.24	0.21	0.21
query19	1.90	1.82	2.02
query20	0.02	0.01	0.01
query21	15.38	0.91	0.58
query22	0.76	0.70	0.67
query23	15.41	1.33	0.59
query24	2.93	1.50	1.57
query25	0.20	0.17	0.12
query26	0.26	0.13	0.13
query27	0.07	0.06	0.06
query28	14.36	0.98	0.42
query29	12.61	4.06	3.33
query30	0.25	0.10	0.07
query31	2.82	0.57	0.40
query32	3.23	0.55	0.46
query33	2.96	3.04	3.09
query34	16.59	5.17	4.46
query35	4.59	4.45	4.53
query36	0.64	0.50	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 106.34 s
Total hot run time: 31.3 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants