Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](cloud-mow) Make delete bitmap cache expired time more reasonable #47295

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

hust-hhb
Copy link
Contributor

after pr #46365, commit mow table may cost 1800s at most (rpc timeout*retry times), so the delete bitmap cache expiration time should be set to 1800s.This cache will be automatically deleted after the loading task is completed. Only when task fails, this cache will be clean in background.

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hust-hhb
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32227 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false

------ Round 1 ----------------------------------
q1	17743	5491	5390	5390
q2	2060	317	178	178
q3	10749	1278	753	753
q4	10521	991	515	515
q5	9295	2430	2230	2230
q6	200	179	138	138
q7	922	776	611	611
q8	9284	1390	1179	1179
q9	5210	4849	4901	4849
q10	6821	2360	1864	1864
q11	485	281	252	252
q12	337	354	215	215
q13	17805	3678	3081	3081
q14	248	227	212	212
q15	512	448	464	448
q16	619	631	603	603
q17	565	894	336	336
q18	7188	6533	6356	6356
q19	1617	973	547	547
q20	320	321	189	189
q21	2791	2267	1956	1956
q22	374	357	325	325
Total cold run time: 105666 ms
Total hot run time: 32227 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5545	5451	5489	5451
q2	236	347	238	238
q3	2280	2652	2316	2316
q4	1463	1830	1390	1390
q5	4371	4768	4794	4768
q6	171	165	131	131
q7	2148	1974	1853	1853
q8	2665	2886	2720	2720
q9	7432	7286	7329	7286
q10	3011	3268	2704	2704
q11	572	534	493	493
q12	712	770	597	597
q13	3528	3886	3317	3317
q14	301	288	303	288
q15	525	458	466	458
q16	644	687	652	652
q17	1238	1734	1274	1274
q18	7982	7599	7372	7372
q19	795	1102	1089	1089
q20	1996	2053	1882	1882
q21	5763	5250	5261	5250
q22	662	609	552	552
Total cold run time: 54040 ms
Total hot run time: 52081 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193108 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false

query1	1330	967	919	919
query2	6358	2056	2010	2010
query3	11001	4663	4341	4341
query4	60878	29098	22978	22978
query5	5614	610	432	432
query6	415	196	176	176
query7	5575	495	296	296
query8	329	248	234	234
query9	8366	2612	2600	2600
query10	474	313	253	253
query11	17916	15039	15484	15039
query12	165	109	118	109
query13	1441	544	426	426
query14	11077	7105	6727	6727
query15	226	205	194	194
query16	7159	632	514	514
query17	1122	737	592	592
query18	1806	383	309	309
query19	195	186	153	153
query20	117	125	107	107
query21	215	125	104	104
query22	4483	4529	4569	4529
query23	34508	33193	33622	33193
query24	5641	2296	2269	2269
query25	514	481	397	397
query26	748	271	158	158
query27	2020	461	340	340
query28	4326	2488	2436	2436
query29	575	571	417	417
query30	217	187	165	165
query31	1020	868	836	836
query32	67	61	55	55
query33	473	379	320	320
query34	732	854	512	512
query35	825	876	748	748
query36	995	1069	970	970
query37	126	97	73	73
query38	4316	4377	4223	4223
query39	1484	1427	1456	1427
query40	201	117	103	103
query41	51	53	51	51
query42	121	104	100	100
query43	511	533	491	491
query44	1284	821	814	814
query45	183	175	173	173
query46	887	1082	669	669
query47	1884	1919	1859	1859
query48	400	413	316	316
query49	734	500	408	408
query50	661	680	398	398
query51	7077	7003	6914	6914
query52	104	100	89	89
query53	225	265	191	191
query54	487	501	427	427
query55	80	82	83	82
query56	268	281	253	253
query57	1193	1195	1189	1189
query58	250	253	238	238
query59	3005	3200	2897	2897
query60	289	265	248	248
query61	120	117	110	110
query62	692	707	658	658
query63	222	186	184	184
query64	2046	1097	654	654
query65	3242	3146	3151	3146
query66	733	386	295	295
query67	16115	15791	15566	15566
query68	5045	818	526	526
query69	498	296	265	265
query70	1241	1117	1157	1117
query71	423	281	257	257
query72	6023	3840	3845	3840
query73	799	744	347	347
query74	9435	9100	8654	8654
query75	3220	3147	2671	2671
query76	3674	1169	773	773
query77	520	362	271	271
query78	10066	10064	9250	9250
query79	3127	817	601	601
query80	1500	531	449	449
query81	546	294	242	242
query82	645	152	125	125
query83	271	164	154	154
query84	285	91	74	74
query85	762	351	319	319
query86	425	282	306	282
query87	4553	4540	4499	4499
query88	4486	2163	2138	2138
query89	400	348	281	281
query90	1547	189	188	188
query91	127	136	106	106
query92	66	56	55	55
query93	2821	874	521	521
query94	874	401	308	308
query95	335	263	245	245
query96	484	610	293	293
query97	2849	2892	2729	2729
query98	222	200	191	191
query99	1274	1360	1233	1233
Total cold run time: 316457 ms
Total hot run time: 193108 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.54 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4110b69826ee04878629733470c19baddfda69a9, data reload: false

query1	0.03	0.03	0.05
query2	0.07	0.03	0.04
query3	0.25	0.07	0.06
query4	1.63	0.10	0.10
query5	0.43	0.43	0.40
query6	1.17	0.65	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.57	0.50	0.50
query10	0.56	0.56	0.55
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.60
query14	2.74	2.86	2.73
query15	0.90	0.83	0.83
query16	0.38	0.39	0.39
query17	1.03	1.06	1.02
query18	0.23	0.22	0.20
query19	1.96	1.84	2.02
query20	0.01	0.01	0.01
query21	15.44	0.91	0.57
query22	0.76	0.77	0.83
query23	15.20	1.46	0.56
query24	3.02	0.80	1.54
query25	0.27	0.09	0.32
query26	0.26	0.15	0.14
query27	0.05	0.06	0.05
query28	14.00	1.07	0.44
query29	12.63	3.92	3.30
query30	0.25	0.09	0.06
query31	2.84	0.62	0.38
query32	3.22	0.58	0.46
query33	3.03	3.03	3.04
query34	16.34	5.10	4.48
query35	4.53	4.43	4.48
query36	0.65	0.48	0.48
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.04	0.03	0.02
query40	0.18	0.13	0.14
query41	0.09	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.94 s
Total hot run time: 30.54 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 41.56% (10834/26068)
Line Coverage: 31.96% (91563/286518)
Region Coverage: 31.09% (46899/150838)
Branch Coverage: 27.18% (23754/87386)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4110b69826ee04878629733470c19baddfda69a9_4110b69826ee04878629733470c19baddfda69a9/report/index.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants