[Subtask]: In master-slave mode, each AMS should automatically senses the optimizer. #3937
[Subtask]: In master-slave mode, each AMS should automatically senses the optimizer. #3937wardlican wants to merge 26 commits intoapache:masterfrom
Conversation
…lave mode is enabled. apache#3845
…lave mode is enabled. apache#3845
…on in master-slave mode. apache#3921
…on in master-slave mode. apache#3921
…on in master-slave mode. apache#3921
…on in master-slave mode. apache#3921
…AMS node for processing. apache#3928
…AMS node for processing. apache#3928
…per in master-slave mode. apache#3928
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions. |
|
Please perform CR on these implementations. |
# Conflicts: # amoro-ams/src/main/java/org/apache/amoro/server/ha/ZkHighAvailabilityContainer.java # amoro-ams/src/main/java/org/apache/amoro/server/table/DefaultTableService.java # amoro-ams/src/main/resources/mysql/upgrade.sql # amoro-ams/src/main/resources/postgres/upgrade.sql # amoro-ams/src/test/java/org/apache/amoro/server/AMSServiceTestBase.java
…troduces a new solution for storing allocation information based on a database.
…troduces a new solution for storing allocation information based on a database.
github-actions
bot
added
type:docs
czy006
left a comment
czy006
left a comment
There was a problem hiding this comment.
Thank you for your contribution; I spent some time reviewing it.
| mapper -> mapper.selectLeasesByService(clusterName, OPTIMIZING_SERVICE)); | ||
| for (HaLeaseMeta lease : leases) { | ||
| // Only include nodes with valid (non-expired) leases | ||
| if (lease.getLeaseExpireTs() != null && lease.getLeaseExpireTs() > currentTime) { |
There was a problem hiding this comment.
Checking only the lease expiration time may return nodes that have just expired but have not yet been cleared.
| amsUrl); | ||
| } | ||
| } catch (TException exception) { | ||
| LOG.error( |
There was a problem hiding this comment.
can we retry to get task? if have exception,maybe lost the task?
| /** | ||
| * Start the assignment service. Only works in master-slave mode and when current node is leader. | ||
| */ | ||
| public void start() { |
There was a problem hiding this comment.
These operations are non-transactional and appear to operate without using locks. Could this lead to inconsistent bucket allocations during Leader switchover?
2 participants
Why are the changes needed?
In master-slave mode, each optimizer needs to communicate with each ams node. Therefore, the registration mode of the optimizer needs to be modified in master-slave mode. The optimizer only reports heartbeats to the master node. Only the master node will maintain whether the optimizer's heartbeat has timed out. The slave node loads optimizer information from the database to detect whether there are any new or deleted optimizers.
Close #3929 .
Brief change log
How was this patch tested?
Add some test cases that check the changes thoroughly including negative and positive cases if possible
Add screenshots for manual tests if appropriate
Run test locally before making a pull request
Documentation