Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](inverted index)Fix load data coredump when encounter exception #45733

Open
wants to merge 2 commits into
base: branch-2.0
Choose a base branch
from

Conversation

qidaye
Copy link
Contributor

@qidaye qidaye commented Dec 20, 2024

What problem does this PR solve?

Issue Number: close #45715

Problem Summary:

Related pr: 45745

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 20, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@qidaye
Copy link
Contributor Author

qidaye commented Dec 20, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49433 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bdb29d6f43f81cbcd01eff99a702cff0c9fa0628, data reload: false

------ Round 1 ----------------------------------
q1	17627	4431	4402	4402
q2	2079	162	158	158
q3	10319	1922	1960	1922
q4	10408	1243	1316	1243
q5	8452	3940	3914	3914
q6	231	123	148	123
q7	2040	1606	1649	1606
q8	9575	2756	2723	2723
q9	13745	10321	9924	9924
q10	8689	3548	3554	3548
q11	420	253	256	253
q12	472	301	307	301
q13	18357	4011	4028	4011
q14	355	334	316	316
q15	517	462	458	458
q16	561	468	453	453
q17	1127	968	942	942
q18	7341	6853	6984	6853
q19	1688	1520	1490	1490
q20	506	311	311	311
q21	4443	4129	4087	4087
q22	505	395	395	395
Total cold run time: 119457 ms
Total hot run time: 49433 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4335	4316	4268	4268
q2	323	229	221	221
q3	4202	4127	4157	4127
q4	2758	2747	2747	2747
q5	7239	7250	7173	7173
q6	239	123	124	123
q7	3229	2831	2878	2831
q8	4464	4516	4501	4501
q9	14261	13855	14050	13855
q10	4231	4269	4316	4269
q11	752	708	689	689
q12	1011	864	850	850
q13	6883	3751	3747	3747
q14	453	434	417	417
q15	502	466	455	455
q16	626	600	576	576
q17	3826	3870	3783	3783
q18	8884	8760	8712	8712
q19	1721	1687	1635	1635
q20	2381	2111	2134	2111
q21	8527	8507	8535	8507
q22	1015	975	961	961
Total cold run time: 81862 ms
Total hot run time: 76558 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.54% (8308/21555)
Line Coverage: 30.23% (68681/227173)
Region Coverage: 29.66% (35365/119242)
Branch Coverage: 25.43% (18186/71526)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bdb29d6f43f81cbcd01eff99a702cff0c9fa0628_bdb29d6f43f81cbcd01eff99a702cff0c9fa0628/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 214233 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bdb29d6f43f81cbcd01eff99a702cff0c9fa0628, data reload: false

query1	982	391	441	391
query2	6557	2245	2123	2123
query3	6921	203	200	200
query4	23422	21715	21661	21661
query5	19749	6542	6572	6542
query6	305	237	236	236
query7	4326	301	311	301
query8	253	249	277	249
query9	3116	2667	2598	2598
query10	456	310	290	290
query11	15800	15119	15101	15101
query12	122	74	75	74
query13	1035	445	450	445
query14	17896	13925	13745	13745
query15	371	222	235	222
query16	6480	280	272	272
query17	1761	967	947	947
query18	895	324	320	320
query19	205	152	152	152
query20	110	101	101	101
query21	188	96	96	96
query22	5225	4959	4767	4767
query23	34323	33654	33679	33654
query24	6987	6314	6375	6314
query25	542	424	424	424
query26	985	170	167	167
query27	2301	298	295	295
query28	6103	2286	2250	2250
query29	2872	2712	2826	2712
query30	243	173	167	167
query31	958	721	747	721
query32	73	65	61	61
query33	449	277	266	266
query34	876	490	477	477
query35	1159	983	963	963
query36	1426	1200	1105	1105
query37	88	63	64	63
query38	3094	2953	2900	2900
query39	1386	1347	1327	1327
query40	209	102	94	94
query41	42	39	38	38
query42	90	84	80	80
query43	622	591	678	591
query44	1187	723	714	714
query45	248	231	230	230
query46	1245	950	972	950
query47	1759	2050	1770	1770
query48	512	439	422	422
query49	623	381	401	381
query50	872	605	611	605
query51	4813	4653	4687	4653
query52	91	86	88	86
query53	227	187	196	187
query54	2670	2502	2481	2481
query55	87	81	83	81
query56	220	212	211	211
query57	1388	1169	1234	1169
query58	217	214	217	214
query59	3299	3455	3342	3342
query60	226	208	211	208
query61	103	99	101	99
query62	832	523	447	447
query63	206	180	180	180
query64	3193	1561	1502	1502
query65	3671	3541	3552	3541
query66	797	421	410	410
query67	18717	15512	15930	15512
query68	8020	628	661	628
query69	517	276	283	276
query70	1545	1531	1362	1362
query71	411	312	320	312
query72	6937	4786	4920	4786
query73	783	320	321	320
query74	6261	5840	5811	5811
query75	4553	3732	3738	3732
query76	4441	1149	1210	1149
query77	553	255	262	255
query78	12680	11830	11763	11763
query79	4967	648	632	632
query80	2557	413	393	393
query81	530	240	239	239
query82	1486	104	95	95
query83	199	135	135	135
query84	262	74	71	71
query85	1406	335	328	328
query86	356	324	276	276
query87	3271	3074	3046	3046
query88	4609	2288	2296	2288
query89	387	289	296	289
query90	1729	218	213	213
query91	172	128	134	128
query92	59	52	54	52
query93	3285	587	587	587
query94	832	212	211	211
query95	2019	2118	2114	2114
query96	637	341	329	329
query97	6603	6531	6466	6466
query98	235	203	208	203
query99	2892	898	847	847
Total cold run time: 313400 ms
Total hot run time: 214233 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bdb29d6f43f81cbcd01eff99a702cff0c9fa0628, data reload: false

query1	0.02	0.02	0.02
query2	0.08	0.02	0.02
query3	0.25	0.05	0.05
query4	1.78	0.10	0.07
query5	0.53	0.54	0.52
query6	1.24	0.62	0.62
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.50	0.48
query10	0.52	0.54	0.53
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.62	0.61
query14	0.78	0.80	0.80
query15	0.79	0.77	0.76
query16	0.37	0.36	0.35
query17	1.02	1.01	0.97
query18	0.24	0.26	0.24
query19	1.95	1.87	1.87
query20	0.02	0.01	0.01
query21	15.45	0.56	0.54
query22	2.07	2.23	1.52
query23	17.38	1.09	1.07
query24	5.19	1.28	1.28
query25	0.31	0.12	0.06
query26	0.58	0.14	0.15
query27	0.03	0.06	0.03
query28	7.62	0.79	0.72
query29	12.61	2.27	2.18
query30	0.65	0.53	0.54
query31	2.82	0.39	0.38
query32	3.34	0.50	0.49
query33	3.08	3.05	3.09
query34	15.26	4.85	4.82
query35	4.87	4.88	4.87
query36	1.07	1.02	1.01
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.17	0.15	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.77 s
Total hot run time: 31.07 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit bdb29d6f43f81cbcd01eff99a702cff0c9fa0628 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

Copy link
Contributor

@zzzxl1993 zzzxl1993 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by anyone and no changes requested.

@qidaye
Copy link
Contributor Author

qidaye commented Dec 20, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49139 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 753fa72e19b496bef39e4a848aa7a063055b6a9f, data reload: false

------ Round 1 ----------------------------------
q1	17628	4419	4355	4355
q2	2068	155	153	153
q3	10347	1892	1933	1892
q4	10371	1248	1322	1248
q5	8881	3942	3934	3934
q6	234	121	124	121
q7	2021	1575	1587	1575
q8	9301	2717	2698	2698
q9	10497	10201	10049	10049
q10	8654	3543	3479	3479
q11	423	235	249	235
q12	461	291	297	291
q13	18377	3956	4062	3956
q14	356	322	321	321
q15	503	457	461	457
q16	522	449	462	449
q17	1131	955	904	904
q18	7228	6755	6862	6755
q19	1661	1559	1528	1528
q20	529	298	308	298
q21	4365	4082	4035	4035
q22	492	412	406	406
Total cold run time: 116050 ms
Total hot run time: 49139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4306	4320	4273	4273
q2	325	226	220	220
q3	4201	4165	4114	4114
q4	2753	2734	2728	2728
q5	7170	7102	7108	7102
q6	243	120	122	120
q7	3269	2859	2855	2855
q8	4362	4449	4515	4449
q9	14466	13835	13925	13835
q10	4188	4270	4251	4251
q11	779	712	746	712
q12	1023	860	843	843
q13	7447	3839	3780	3780
q14	467	432	437	432
q15	504	442	456	442
q16	645	609	589	589
q17	3870	3745	3876	3745
q18	9013	8909	8682	8682
q19	1720	1693	1625	1625
q20	2357	2126	2142	2126
q21	8523	8404	8445	8404
q22	992	975	947	947
Total cold run time: 82623 ms
Total hot run time: 76274 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.56% (8312/21555)
Line Coverage: 30.25% (68720/227173)
Region Coverage: 29.68% (35392/119242)
Branch Coverage: 25.44% (18199/71526)
Coverage Report: http://coverage.selectdb-in.cc/coverage/753fa72e19b496bef39e4a848aa7a063055b6a9f_753fa72e19b496bef39e4a848aa7a063055b6a9f/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 212475 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 753fa72e19b496bef39e4a848aa7a063055b6a9f, data reload: false

query1	942	427	378	378
query2	6531	2295	2116	2116
query3	6927	203	200	200
query4	23292	21582	21447	21447
query5	19735	6523	6539	6523
query6	289	222	234	222
query7	4334	324	321	321
query8	260	264	270	264
query9	3068	2657	2590	2590
query10	467	321	307	307
query11	15381	14904	14984	14904
query12	134	76	74	74
query13	1038	456	445	445
query14	17378	13617	13569	13569
query15	411	226	227	226
query16	6488	273	258	258
query17	1774	926	884	884
query18	890	320	317	317
query19	224	153	159	153
query20	96	96	104	96
query21	195	96	93	93
query22	5259	5032	4969	4969
query23	34134	33361	33535	33361
query24	7670	6342	6339	6339
query25	511	443	412	412
query26	1277	171	161	161
query27	2281	304	306	304
query28	6104	2280	2269	2269
query29	2857	2819	2878	2819
query30	240	169	166	166
query31	980	723	742	723
query32	70	62	64	62
query33	458	262	256	256
query34	867	507	506	506
query35	1142	963	966	963
query36	1284	1110	1287	1110
query37	171	66	62	62
query38	3105	2953	2873	2873
query39	1355	1322	1339	1322
query40	310	101	99	99
query41	41	38	40	38
query42	87	90	90	90
query43	645	570	637	570
query44	1175	712	717	712
query45	245	236	235	235
query46	1234	988	995	988
query47	1939	1553	1851	1553
query48	516	415	414	414
query49	644	377	384	377
query50	869	639	625	625
query51	4807	4671	4790	4671
query52	98	83	88	83
query53	232	188	194	188
query54	2676	2471	2506	2471
query55	87	89	86	86
query56	230	214	210	210
query57	1371	1111	1152	1111
query58	224	218	213	213
query59	3695	3046	3351	3046
query60	228	211	197	197
query61	98	94	118	94
query62	836	494	437	437
query63	209	187	178	178
query64	3520	1580	1455	1455
query65	3616	3568	3568	3568
query66	788	428	397	397
query67	15693	15203	14884	14884
query68	11319	659	644	644
query69	521	262	287	262
query70	2046	1407	1408	1407
query71	421	305	319	305
query72	6762	4879	4921	4879
query73	771	321	324	321
query74	6320	5820	5887	5820
query75	5446	3645	3645	3645
query76	6840	1126	1212	1126
query77	1196	261	271	261
query78	12652	11904	18664	11904
query79	3896	648	662	648
query80	751	404	404	404
query81	490	249	238	238
query82	295	106	103	103
query83	176	136	137	136
query84	262	71	72	71
query85	881	337	328	328
query86	342	300	330	300
query87	3253	3045	3024	3024
query88	3326	2300	2285	2285
query89	358	302	296	296
query90	1945	217	222	217
query91	169	129	126	126
query92	61	49	53	49
query93	945	543	550	543
query94	716	211	212	211
query95	2108	2075	2089	2075
query96	648	341	324	324
query97	6629	6394	6501	6394
query98	243	212	200	200
query99	2963	892	875	875
Total cold run time: 310483 ms
Total hot run time: 212475 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.41 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 753fa72e19b496bef39e4a848aa7a063055b6a9f, data reload: false

query1	0.02	0.03	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.78	0.06	0.08
query5	0.52	0.53	0.51
query6	1.23	0.62	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.48	0.48
query10	0.55	0.54	0.52
query11	0.12	0.09	0.09
query12	0.12	0.10	0.09
query13	0.62	0.61	0.62
query14	0.77	0.81	0.80
query15	0.78	0.77	0.76
query16	0.38	0.38	0.40
query17	1.00	0.99	1.01
query18	0.24	0.23	0.25
query19	1.91	1.80	1.89
query20	0.02	0.01	0.01
query21	15.44	0.55	0.55
query22	2.08	2.71	2.37
query23	17.24	1.01	0.99
query24	5.34	0.78	1.56
query25	0.37	0.08	0.07
query26	0.55	0.16	0.15
query27	0.05	0.04	0.05
query28	7.43	0.73	0.72
query29	12.78	2.33	2.35
query30	0.67	0.60	0.54
query31	2.80	0.37	0.39
query32	3.38	0.50	0.50
query33	3.10	3.04	3.08
query34	15.26	4.78	4.82
query35	4.85	4.87	4.86
query36	1.04	1.03	1.03
query37	0.06	0.05	0.06
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.07	0.01	0.02
query42	0.03	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 103.72 s
Total hot run time: 31.41 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 753fa72e19b496bef39e4a848aa7a063055b6a9f with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.2 seconds inserted 10000000 Rows, about 450K ops/s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants