Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](exchange)fix exchange sink buffer does not update total_queue_size when EOF. #47322

Open
wants to merge 1 commit into
base: branch-3.0
Choose a base branch
from

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Jan 22, 2025

What problem does this PR solve?

pick part from #47312

#41602
EOF clears _instance_to_package_queue but does not update total_queue_size, causing incorrect judgments that rely on total_queue_size.

UT

mock transmit_blockv2 dest ins id :1
mock transmit_blockv2 dest ins id :2
mock transmit_blockv2 dest ins id :3
queue size : 6
each queue size : 
Instance: 2, queue size: 2
Instance: 1, queue size: 2
Instance: 3, queue size: 2

queue size : 6 // error size 
each queue size :
Instance: 2, queue size: 0
Instance: 1, queue size: 2
Instance: 3, queue size: 2

mock transmit_blockv2 dest ins id :1
mock transmit_blockv2 dest ins id :1
mock transmit_blockv2 dest ins id :3
mock transmit_blockv2 dest ins id :3

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Mryange Mryange requested a review from dataroaring as a code owner January 22, 2025 10:45
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange
Copy link
Contributor Author

Mryange commented Jan 22, 2025

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40814 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0307872937acdb3ee21d61622381f7175706dba6, data reload: false

------ Round 1 ----------------------------------
q1	17591	7354	7312	7312
q2	2063	170	160	160
q3	10554	1157	1161	1157
q4	10232	714	753	714
q5	7744	2812	2827	2812
q6	232	152	147	147
q7	963	618	615	615
q8	9392	1964	1974	1964
q9	6677	6402	6480	6402
q10	7866	2286	2324	2286
q11	458	263	259	259
q12	402	210	219	210
q13	17777	2997	2972	2972
q14	255	213	206	206
q15	562	531	547	531
q16	671	615	613	613
q17	971	586	519	519
q18	7356	6659	6696	6659
q19	1398	1143	1008	1008
q20	474	199	205	199
q21	3998	3302	3084	3084
q22	1081	995	985	985
Total cold run time: 108717 ms
Total hot run time: 40814 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7218	7243	7231	7231
q2	324	237	229	229
q3	2905	3041	2995	2995
q4	2071	1842	1845	1842
q5	5713	5769	5736	5736
q6	220	136	136	136
q7	2210	1816	1805	1805
q8	3351	3505	3499	3499
q9	8870	8889	8884	8884
q10	3583	3571	3564	3564
q11	586	501	497	497
q12	828	583	654	583
q13	4625	3186	3129	3129
q14	296	274	271	271
q15	563	523	522	522
q16	717	672	682	672
q17	1864	1602	1584	1584
q18	8196	7694	7610	7610
q19	2093	1588	1641	1588
q20	2039	1801	1812	1801
q21	5135	5158	5207	5158
q22	1083	999	1022	999
Total cold run time: 64490 ms
Total hot run time: 60335 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191245 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0307872937acdb3ee21d61622381f7175706dba6, data reload: false

query1	998	367	360	360
query2	6509	2086	2012	2012
query3	6702	215	227	215
query4	34126	23494	23399	23399
query5	4421	456	436	436
query6	266	173	184	173
query7	4616	308	312	308
query8	283	215	215	215
query9	9679	2687	2674	2674
query10	490	262	272	262
query11	18196	15328	15199	15199
query12	153	104	106	104
query13	1648	433	410	410
query14	9992	7158	7755	7158
query15	260	183	183	183
query16	8052	445	434	434
query17	1694	569	563	563
query18	2019	334	317	317
query19	371	162	160	160
query20	119	110	110	110
query21	208	104	106	104
query22	4536	4199	4337	4199
query23	34881	33727	34025	33727
query24	11593	2866	2907	2866
query25	684	402	388	388
query26	1598	167	169	167
query27	2769	341	348	341
query28	7890	2456	2425	2425
query29	979	423	416	416
query30	328	161	163	161
query31	1037	819	823	819
query32	96	60	61	60
query33	776	292	306	292
query34	934	495	521	495
query35	892	717	700	700
query36	1097	953	918	918
query37	133	68	71	68
query38	4142	3951	3869	3869
query39	1477	1414	1445	1414
query40	287	100	99	99
query41	52	49	50	49
query42	114	102	104	102
query43	549	486	492	486
query44	1327	803	804	803
query45	186	166	169	166
query46	1140	711	707	707
query47	1948	1777	1828	1777
query48	463	378	368	368
query49	1239	386	379	379
query50	816	411	411	411
query51	7144	7293	6980	6980
query52	102	93	91	91
query53	261	183	186	183
query54	1205	459	449	449
query55	77	79	79	79
query56	272	239	245	239
query57	1335	1117	1084	1084
query58	243	210	229	210
query59	3256	2918	2875	2875
query60	282	260	290	260
query61	112	108	108	108
query62	896	675	674	674
query63	215	197	191	191
query64	5219	670	639	639
query65	3295	3179	3236	3179
query66	1428	316	331	316
query67	15834	15578	15536	15536
query68	3561	603	590	590
query69	419	269	275	269
query70	1195	1071	1029	1029
query71	335	263	267	263
query72	6251	4027	4085	4027
query73	742	346	349	346
query74	9990	9041	9213	9041
query75	3336	2650	2639	2639
query76	2504	1113	1003	1003
query77	401	279	283	279
query78	10398	9598	9642	9598
query79	1094	587	592	587
query80	782	434	427	427
query81	507	241	238	238
query82	1151	132	121	121
query83	245	145	152	145
query84	232	81	76	76
query85	1095	295	279	279
query86	321	302	291	291
query87	4438	4502	4345	4345
query88	3374	2402	2364	2364
query89	397	298	298	298
query90	2079	196	193	193
query91	187	148	147	147
query92	64	52	53	52
query93	1090	539	527	527
query94	796	285	295	285
query95	361	257	259	257
query96	608	281	284	281
query97	3292	3171	3185	3171
query98	217	201	197	197
query99	1522	1319	1441	1319
Total cold run time: 298925 ms
Total hot run time: 191245 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0307872937acdb3ee21d61622381f7175706dba6, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.54	0.50	0.51
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.03
query9	0.56	0.51	0.51
query10	0.56	0.56	0.56
query11	0.15	0.10	0.11
query12	0.15	0.12	0.14
query13	0.61	0.60	0.59
query14	2.95	2.95	2.96
query15	0.90	0.83	0.82
query16	0.38	0.37	0.37
query17	1.02	1.00	0.99
query18	0.23	0.23	0.22
query19	1.83	1.78	2.03
query20	0.02	0.01	0.01
query21	15.35	0.61	0.58
query22	2.52	2.82	1.99
query23	17.04	0.96	0.77
query24	3.00	1.17	1.49
query25	0.11	0.13	0.20
query26	0.60	0.14	0.14
query27	0.04	0.04	0.04
query28	9.95	1.10	1.06
query29	12.56	3.32	3.36
query30	0.25	0.06	0.06
query31	2.87	0.39	0.37
query32	3.24	0.46	0.46
query33	2.99	3.01	3.04
query34	16.79	4.47	4.54
query35	4.48	4.56	4.52
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.04	0.02	0.03
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.01 s
Total hot run time: 33.38 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants