HDFS-17690. Avoid redundant EC reconstruction tasks after timeouts #7231

junegunn · 2024-12-18T05:01:41Z

Description of PR

Avoid processing redundant BlockECReconstructionInfo from the queue. Check again if the task is really needed just before we dispatch the tasks to the datanodes.

How was this patch tested?

Tested on a 3.4.1 cluster with 21 datanodes. Shutted down 2 datanodes holding 4~5TB of data and observed the reconstruction progress.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

hadoop-yetus · 2024-12-18T09:37:19Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	6m 40s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	23m 14s		trunk passed
+1 💚	compile	0m 43s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 40s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 39s		trunk passed
+1 💚	mvnsite	0m 44s		trunk passed
+1 💚	javadoc	0m 44s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 6s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 40s		trunk passed
+1 💚	shadedclient	19m 54s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 35s		the patch passed
+1 💚	compile	0m 36s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 36s		the patch passed
+1 💚	compile	0m 37s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 37s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 30s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 131 unchanged - 0 fixed = 135 total (was 131)
+1 💚	mvnsite	0m 36s		the patch passed
+1 💚	javadoc	0m 30s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 0s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 35s		the patch passed
+1 💚	shadedclient	19m 38s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	192m 39s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		274m 32s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission
	hadoop.hdfs.server.blockmanagement.TestDatanodeManager
	hadoop.hdfs.TestDecommissionWithBackoffMonitor

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7231/1/artifact/out/Dockerfile
GITHUB PR	#7231
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 9f99b4ac917c 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8677678`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7231/1/testReport/
Max. process+thread count	4866 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7231/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

junegunn · 2024-12-19T02:44:48Z

Let me look into the test failures.

The purpose of this change is to make the existing testComputeReconstructedTaskNum pass, which uses a mock object that behaves differently to the actual implementation (returns a 0-length list instead of null).

junegunn · 2024-12-19T03:58:19Z

18:37:24                Reason | Tests
18:37:24   Failed junit tests  |  hadoop.hdfs.TestDecommission 
18:37:24                       |  hadoop.hdfs.server.blockmanagement.TestDatanodeManager 
18:37:24                       |  hadoop.hdfs.TestDecommissionWithBackoffMonitor

TestDecommission: Passes locally
TestDecommissionWithBackoffMonitor: Passes locally
TestDatanodeManager: Passes with 151a518, an unnecessary change to satisfy a mock verification of the failed test case.

hadoop-yetus · 2024-12-19T08:22:36Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	7m 41s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 1 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	24m 1s		trunk passed
+1 💚	compile	0m 45s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 41s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 38s		trunk passed
+1 💚	mvnsite	0m 43s		trunk passed
+1 💚	javadoc	0m 43s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 6s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 49s		trunk passed
+1 💚	shadedclient	20m 19s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 37s		the patch passed
+1 💚	compile	0m 39s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 39s		the patch passed
+1 💚	compile	0m 31s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 27s	/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 131 unchanged - 0 fixed = 135 total (was 131)
+1 💚	mvnsite	0m 39s		the patch passed
+1 💚	javadoc	0m 34s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	1m 1s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 39s		the patch passed
+1 💚	shadedclient	20m 17s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
-1 ❌	unit	191m 58s	/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt	hadoop-hdfs in the patch passed.
+1 💚	asflicense	0m 30s		The patch does not generate ASF License warnings.
		276m 34s

Reason	Tests
Failed junit tests	hadoop.hdfs.TestDecommission
	hadoop.hdfs.TestDecommissionWithBackoffMonitor

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7231/2/artifact/out/Dockerfile
GITHUB PR	#7231
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux accb65c9f12c 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `151a518`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7231/2/testReport/
Max. process+thread count	4612 (vs. ulimit of 5500)
modules	C: hadoop-hdfs-project/hadoop-hdfs U: hadoop-hdfs-project/hadoop-hdfs
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7231/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

HDFS-17690. Avoid redundant EC reconstruction tasks after timeouts

8677678

github-actions bot added HDFS trunk labels Dec 18, 2024

Make TestDatanodeManager.testComputeReconstructedTaskNum pass

151a518

The purpose of this change is to make the existing testComputeReconstructedTaskNum pass, which uses a mock object that behaves differently to the actual implementation (returns a 0-length list instead of null).

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HDFS-17690. Avoid redundant EC reconstruction tasks after timeouts #7231

HDFS-17690. Avoid redundant EC reconstruction tasks after timeouts #7231

junegunn commented Dec 18, 2024

hadoop-yetus commented Dec 18, 2024

junegunn commented Dec 19, 2024

junegunn commented Dec 19, 2024

hadoop-yetus commented Dec 19, 2024

HDFS-17690. Avoid redundant EC reconstruction tasks after timeouts #7231

Are you sure you want to change the base?

HDFS-17690. Avoid redundant EC reconstruction tasks after timeouts #7231

Conversation

junegunn commented Dec 18, 2024

Description of PR

How was this patch tested?

For code changes:

hadoop-yetus commented Dec 18, 2024

junegunn commented Dec 19, 2024

junegunn commented Dec 19, 2024

hadoop-yetus commented Dec 19, 2024