feat: Supports node black list for load balance

[lengyuexuexuan]

What problem does this PR solve?

#1976

In addition this, solve two issues for cluster_balance_policy.
1.

incubator-pegasus/src/meta/cluster_balance_policy.cpp

Lines 201 to 202 in cd1682d

	cluster_info.apps_info.emplace(kv.first, std::move(info));
	cluster_info.apps_skew[kv.first] = get_skew(info.replicas_count);

std::move(info) is before use the variable info, causes the skew is wrong.

incubator-pegasus/src/meta/load_balance_policy.h

Line 113 in cd1682d

std::set<app_id> _balancer_ignored_apps;

incubator-pegasus/src/meta/greedy_load_balancer.h

Lines 77 to 78 in cd1682d

	std::unique_ptr<load_balance_policy> _app_balance_policy;
	std::unique_ptr<load_balance_policy> _cluster_balance_policy;

The variable _balancer_ignored_apps is not static, that causes _app_balance_policy and _cluster_balance_policy have separate _balancer_ignored_apps. So, when we set _balancer_ignored_apps, it only takes effect on _app_balance_policy.

Both issues will be fixed in this pr.

What is changed and how does it work?

• command

meta.lb.ignored_nodes_list <get|set|clear> [node_addr1,nodes_addr2..]
Supports get, set, and clear commands.
The number of blacklisted nodes must not exceed the number of alive_nodes minus 2, otherwise balancing will not be possible.

• app_balance_policy

  • Move primary
    No increase or decrease in node slicing is involved, so no restriction is applied.
  • copy primary and copy secondary
    The balancing strategy is the same for both phases, so the restriction method is the same.

  Sort the nodes by the number of primary replica from smallest to largest to get pri_queue
  On the pri_queue, id_min always points to the head node of the pri_queue and id_max always points to the tail node of the pri_queue, 
  as shown below.
   +------+------+------+------+------+------+------+------+
   |                                                       |
   V                                                       V
  id_min                                                  id_max
  For all replicas on the current id_max, find their corresponding disks and obtain their disk loads, select the disk with the highest load and its corresponding primary replica, and perform the relocation.
  +1/-1 for the current number of primaries pointed to by id_min/id_max, respectively. reorder and loop through the above steps until the number of primaries on the id_min node >= N/M, at which point balance is reached
  

  The difference between copy primary and copy secondary is simply that the queue for copy primary is sorted based on the
  number of primary slices of the table on each node. Copy secondary is sorted based on the number of all slices of the table
  on each node.
  Therefore, it is sufficient to exclude the blacklisted nodes when choosing id_min/id_max.

• cluster_balance_policy

  • Move primary:
    No increase or decrease in node slicing is involved, so no restriction is applied.
  • copy primary and copy secondary
    The balancing strategy is the same for both phases, so the restriction method is the same.
    The difference between copy primary and copy secondary is simply that the number of slices computed for copy primary is the primary slice and the number of slices computed for copy secondary is the slave slice (excluding the primary slice).
    The strategy to implement node blacklisting is:
    1. Ignore the blacklisted on nodes when calculating skew. That is, when calculating both app_skew and server_skew, the nodes on the node blacklist are not counted, and it is sufficient for the remaining nodes to reach balance.
    2. When calculating cluster_min_count_nodes, cluster_max_count_nodes, app_min_count_nodes, app_max_count_nodes, ignore the blacklisted nodes.
    3. Similar to app_balance, i.e., ignore nodes when selecting max and min nodes.

Checklist

Tests

• Unit test

// 测试app balance
export GTEST_FILTER=meta.app_balancer_nodes_blacklist_test
./run.sh test -m dsn.meta.test

// 测试cluster balance
export GTEST_FILTER=meta.cluster_balancer_nodes_blacklist_test
./run.sh test -m dsn.meta.test

• Manual test (add detailed scripts or steps below)

1. Building unbalanced clusters with onebox.
   Use node restart and the command remote_command -t meta-server meta.lb.assign_secondary_black_list $address_list
2. set node blacklist
3. use command set_meta_level lively.

• app_balance_policy
  The initial state of the cluster is:
  
  Set 172.17.0.2:34801, 172.17.0.2:34806 as blacklisted, and then load-balance with a termination state of:
  
  It can be seen that the number of slices for two nodes, 172.17.0.2:34801 and 172.17.0.2:34806, did not change, and the other four nodes reached a balanced state. After clear ignored_node_list, perform balance, the result is:
  
• cluster_balance_policy
  The initial state of the cluster is:
  
  Set 172.17.0.2:34801, 172.17.0.2:34806 as blacklisted, and then load-balance with a termination state of:
  
  It can be seen that the number of slices for two nodes, 172.17.0.2:34801 and 172.17.0.2:34806, did not change, and the other four nodes reached a balanced state. After clear ignored_node_list, perform balance, the result is: