HBASE-29236: Add Support for Dynamic Configuration at the Coprocessor Level #6931

kgeisz · 2025-04-23T23:06:17Z

https://issues.apache.org/jira/browse/HBASE-29236

This pull request adds support for Dynamic Configuration at the coprocessor level and makes the hbase.global.readonly.enabled config variable dynamically configurable. This variable is used with the ReadOnlyController coprocessor. In order to make this class dynamically configurable, we have it implement the ConfigurationObserver interface. We also need to register this observer with the ConfigurationManager, so the ReadOnlyController is registered to HMaster's, HRegionServer's, and HRegion's ConfigurationManager. Also, since the ReadOnlyController is a coprocessor, we need to use the CoprocessorHost in order to get the coprocessor and register it with the ConfigurationManager.

hbase.global.readonly.enabled can be configured dynamically in the following way:

Add hbase.global.readonly.enabled to hbase-site.xml:

<property>
  <name>hbase.global.readonly.enabled</name>
  <value>false</value>
</property>

Add the ReadOnlyController coprocessor to the master, region server, and region by adding the following to hbase-site.xml:

<property>
  <name>hbase.coprocessor.master.classes</name>
  <value>org.apache.hadoop.hbase.security.access.ReadOnlyController</value>
</property>
<property>
  <name>hbase.coprocessor.regionserver.classes</name>
  <value>org.apache.hadoop.hbase.security.access.ReadOnlyController</value>
</property>
<property>
  <name>hbase.coprocessor.region.classes</name>
  <value>org.apache.hadoop.hbase.security.access.ReadOnlyController</value>
</property>

Start HBase
Change hbase.global.readonly.enabled from false to true in hbase-site.xml and save the file.
In the HBase shell, run update_all_config.
You should see the following log message for the master, region servers(s), and region(s):

2025-04-23T15:23:59,008 INFO  [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16000] access.ReadOnlyController: Config hbase.global.readonly.enabled has been dynamically changed to true

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java

anmolnar

Patch looks good to me. Thanks @kgeisz !
I think this is the point that you should work together with @sharmaar12 and add unit test which starts up a non-RO mini cluster, dynamically changes it to RO mode and validate some mutatation commands.

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java

Kota-SH

LGTM, just a couple of nits.
+1 for the unit tests.

hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java

kgeisz · 2025-05-06T00:56:35Z

@anmolnar, I have added some unit test cases. I also did a couple more things:

I had to register the ReadOnlyController to HRegion's ConfigurationManager in order for an exception to get thrown when a Put command is performed with hbase.global.readonly.enabled set to true.
I removed the WALEdit checks inside of prePut() and preBatchMutate() for ReadOnlyController.java.

anmolnar

lgtm. Thanks for the tests!

anmolnar · 2025-05-08T21:46:42Z

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java

-    if (edit.isMetaEdit() || edit.isEmpty()) {
-      return;
-    }


So, you don't need these checks anymore, because you start the mini cluster in R/W mode in your tests. Does that mean a real cluster can be started in R/O mode, have you verified that?

@anmolnar, I cannot start HBase in Read-Only mode with these current changes.

@anmolnar, I made it so prePut() and preBatchMutate() will skip the internalReadOnlyGuard() check if the table is a system table, rather than if the WALEdit is empty or a meta edit. I also added a unit test class that verifies HBase can be started in Read-Only mode.

Change-Id: If6e6104eca7937a0a478cb319bafefa5aefec708

…as actually changed Change-Id: I552c429df8f65e843de554fd8624854821b75c4e

…rs() Change-Id: If19fb7d206f22d0950e97d6b65b233092e11a13a

…led with Region coprocessors Change-Id: Ie34bbd973fab001bde86c29c35e7a3d89db395b8

Change-Id: I0c334c5fa85c0ba1b315705bd453584a96664009

….global.readonly.enabled Change-Id: If688376a44eceebb7e2829781e71a30b9985eb6b

Change-Id: Idf2432c7752a53c229d0d0dcaadb3582087635fe

Change-Id: I56f866ea20e585606d1d9102f815f008863e7c67

Change-Id: I5f4f57b63a8707d00dc67abc912abf4d9f1a7959

…tchMutate() when working with a system table Change-Id: I84019e536dc2813d68eb480a12791787ddfe4cf3

…nly mode Change-Id: I64904d94bad6ada43c4a49fe6b2eaf3193db34e0

anmolnar

great stuff! thanks @kgeisz !

NihalJain · 2025-05-27T16:28:59Z

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestReadOnlyController.java

+      TEST_UTIL.deleteTable(TEST_TABLE);
+      connection.close();
+      TEST_UTIL.shutdownMiniCluster();
+      throw new RuntimeException(e);


I hope this would not cause the whole test pipeline to fail?

hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java

NihalJain · 2025-05-27T16:35:10Z

Maybe it is worth pondering if we should rename this JIRA to highlight the fact that it actually also adds support for dynamic conf at coprocessor level, which may expose it to others leading to several other use cases, with maybe R/O flag being an example implementation? I am fine even if we break this into 2 jiras to expose the same.

Anyways overall LGTM, +1 from me as well. Nice work!

kgeisz · 2025-05-28T16:50:26Z

@NihalJain, thanks for the feedback! I updated the name of the Jira ticket and added some info to the ticket's description. I also updated the name of the PR to match the Jira ticket.

Apache-HBase · 2025-05-28T22:31:38Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 31s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	hbaseanti	0m 0s		Patch does not have any anti-patterns.
			_ HBASE-29081 Compile Tests _
+0 🆗	mvndep	0m 13s		Maven dependency ordering for branch
+1 💚	mvninstall	3m 17s		HBASE-29081 passed
+1 💚	compile	7m 53s		HBASE-29081 passed
+1 💚	checkstyle	1m 10s		HBASE-29081 passed
+1 💚	spotbugs	8m 58s		HBASE-29081 passed
+1 💚	spotless	0m 47s		branch has no errors when running spotless:check.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 11s		Maven dependency ordering for patch
+1 💚	mvninstall	3m 6s		the patch passed
+1 💚	compile	7m 52s		the patch passed
+1 💚	javac	7m 52s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	1m 12s	/results-checkstyle-root.txt	root: The patch generated 3 new + 29 unchanged - 0 fixed = 32 total (was 29)
+1 💚	spotbugs	9m 7s		the patch passed
+1 💚	hadoopcheck	11m 58s		Patch does not cause any errors with Hadoop 3.3.6 3.4.0.
+1 💚	spotless	0m 44s		patch has no errors when running spotless:check.
			_ Other Tests _
+1 💚	asflicense	0m 20s		The patch does not generate ASF License warnings.
		65m 4s

Subsystem	Report/Notes
Docker	ClientAPI=1.43 ServerAPI=1.43 base: https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6931/13/artifact/yetus-general-check/output/Dockerfile
GITHUB PR	#6931
JIRA Issue	HBASE-29236
Optional Tests	dupname asflicense javac spotbugs checkstyle codespell detsecrets compile hadoopcheck hbaseanti spotless
uname	Linux 8044e1bfec8e 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/hbase-personality.sh
git revision	HBASE-29081 / `faf86e3`
Default Java	Eclipse Adoptium-17.0.11+9
Max. process+thread count	190 (vs. ulimit of 30000)
modules	C: hbase-server . U: .
Console output	https://ci-hbase.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-6931/13/console
versions	git=2.34.1 maven=3.9.8 spotbugs=4.7.3
Powered by	Apache Yetus 0.15.0 https://yetus.apache.org

This message was automatically generated.

anmolnar · 2025-05-29T12:41:31Z

Build timed out, but it was successful, so I merge the patch regardless. Thanks @kgeisz !

… Level (#6931) Co-authored-by: Andor Molnar <[email protected]> Co-authored-by: Anuj Sharma <[email protected]>

kgeisz force-pushed the HBASE-29236-add-readonly-dynamic-config branch from 8bda4a3 to 37a7919 Compare April 23, 2025 23:19

anmolnar reviewed Apr 24, 2025

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java Show resolved Hide resolved

This comment has been minimized.

Sign in to view

wchevreuil approved these changes Apr 24, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

anmolnar requested changes Apr 29, 2025

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/security/access/ReadOnlyController.java Outdated Show resolved Hide resolved

kgeisz requested a review from anmolnar April 29, 2025 18:27

This comment has been minimized.

Sign in to view

Kota-SH reviewed Apr 30, 2025

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java Show resolved Hide resolved

kgeisz force-pushed the HBASE-29236-add-readonly-dynamic-config branch from 5fa6a0b to b9a05d8 Compare May 6, 2025 00:24

This comment has been minimized.

Sign in to view

anmolnar approved these changes May 8, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

anmolnar force-pushed the HBASE-29081 branch from b1cb276 to aaa0c9e Compare May 16, 2025 15:45

anmolnar and others added 4 commits May 19, 2025 14:50

HBASE-29083. Allow test to update hbase:meta table

97f2e55

HBASE-29083. Spotless apply

dc9342c

Refactor code to have only passing tests

8cb6784

Apply spotless

3d3f5e1

kgeisz added 9 commits May 19, 2025 14:50

HBASE-29081: Update warning message

0517f0b

Change-Id: If6e6104eca7937a0a478cb319bafefa5aefec708

HBASE-29081: Only update hbase.global.readonly.enabled if its value h…

0e91088

…as actually changed Change-Id: I552c429df8f65e843de554fd8624854821b75c4e

HBASE-29081: Add method documentation to registerConfigurationObserve…

6a157ae

…rs() Change-Id: If19fb7d206f22d0950e97d6b65b233092e11a13a

HBASE-29081: Add dynamic configuration for hbase.global.readonly.enab…

7c58405

…led with Region coprocessors Change-Id: Ie34bbd973fab001bde86c29c35e7a3d89db395b8

HBASE-29081: Remove WALEdit checks from prePut() and preBatchMutate()

fab41ca

Change-Id: I0c334c5fa85c0ba1b315705bd453584a96664009

HBASE-29081: Add unit test cases for the HBase Put command with hbase…

0bbfb83

….global.readonly.enabled Change-Id: If688376a44eceebb7e2829781e71a30b9985eb6b

HBASE-29081: Update comments

4c68b5b

Change-Id: Idf2432c7752a53c229d0d0dcaadb3582087635fe

Move static methods above test methods

c470095

Change-Id: I56f866ea20e585606d1d9102f815f008863e7c67

Add comment

035cfdb

Change-Id: I5f4f57b63a8707d00dc67abc912abf4d9f1a7959

kgeisz force-pushed the HBASE-29236-add-readonly-dynamic-config branch from 5e2854e to 035cfdb Compare May 19, 2025 21:51

This comment has been minimized.

Sign in to view

kgeisz added 2 commits May 21, 2025 14:48

HBASE-29236: Skip internalReadOnlyGuard() check in prePut() and preBa…

fc41b3b

…tchMutate() when working with a system table Change-Id: I84019e536dc2813d68eb480a12791787ddfe4cf3

HBASE-29236: Add unit test for proving HBase can be started in Read-O…

faf86e3

…nly mode Change-Id: I64904d94bad6ada43c4a49fe6b2eaf3193db34e0

This comment has been minimized.

Sign in to view

kgeisz requested a review from anmolnar May 22, 2025 17:59

anmolnar approved these changes May 27, 2025

View reviewed changes

NihalJain reviewed May 27, 2025

View reviewed changes

hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Show resolved Hide resolved

NihalJain approved these changes May 27, 2025

View reviewed changes

Kota-SH approved these changes May 27, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

kgeisz changed the title ~~HBASE-29236: Add ability to flip R/O flag runtime~~ HBASE-29236: Add Support for Dynamic Configuration at the Coprocessor Level May 28, 2025

anmolnar merged commit e34ec5f into apache:HBASE-29081 May 29, 2025
1 check failed

anmolnar added a commit that referenced this pull request Aug 18, 2025

HBASE-29236: Add Support for Dynamic Configuration at the Coprocessor…

ae8fd61

… Level (#6931) Co-authored-by: Andor Molnar <[email protected]> Co-authored-by: Anuj Sharma <[email protected]>

anmolnar added a commit that referenced this pull request Sep 9, 2025

HBASE-29236: Add Support for Dynamic Configuration at the Coprocessor…

f6c712d

… Level (#6931) Co-authored-by: Andor Molnar <[email protected]> Co-authored-by: Anuj Sharma <[email protected]>

HBASE-29236: Add Support for Dynamic Configuration at the Coprocessor Level #6931

HBASE-29236: Add Support for Dynamic Configuration at the Coprocessor Level #6931

Uh oh!

Conversation

kgeisz commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

Kota-SH left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kgeisz commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

anmolnar May 8, 2025

Choose a reason for hiding this comment

Uh oh!

kgeisz May 19, 2025

Choose a reason for hiding this comment

Uh oh!

kgeisz May 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

anmolnar left a comment

Choose a reason for hiding this comment

Uh oh!

NihalJain May 27, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NihalJain commented May 27, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

kgeisz commented May 28, 2025

Uh oh!

Apache-HBase commented May 28, 2025

Uh oh!

Uh oh!

anmolnar commented May 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kgeisz commented Apr 23, 2025 •

edited

Loading

kgeisz commented May 6, 2025 •

edited

Loading

kgeisz May 22, 2025 •

edited

Loading