[BUG] When kube-ovn-controller restarts, need takes a long time to start process queue tasks

[cmdy]

Kube-OVN Version

v1.12.29

Kubernetes Version

v1.28.11

Operation-system/Kernel Version

"CentOS Linux 7 (Core)" 5.10.0-228.2410.el7.bzl.x86_64

Description

1. when kube-ovn-controller restarts, need takes a long time(1 hour+) to start process queue tasks in a large cluster(2000+Node,15w+Pod,100+subnet)
2. work queue depth monitor
   

Steps To Reproduce

1. creating a large cluster(2000+Node,15w+Pod,100+subnet)
2. restart kube-ovn-controller pod

Current Behavior

need takes a long time(1 hour+) to start queue tasks process

Expected Behavior

fast to start queue tasks process

[dosubot]

The issue you're experiencing with the kube-ovn-controller taking a long time to start processing queue tasks after a restart in a large cluster is likely due to two main factors:

1. Slow Policy Route Addition: Each policy route addition operation is taking 1-3 seconds, which significantly delays the processing of the entire queue.

2. Duplicate Operations: There are duplicate operations for the same node and subnet combinations, resulting in more operations than expected, which further slows down the process.

These factors contribute to high work queue latency and slow queue depth reduction, causing the processing to take several hours instead of minutes ^[1].

To improve the processing speed, you can increase the WorkerNum parameter using the --worker-num flag, which controls the parallelism of each worker. By default, it is set to 3, but increasing this number can help process more tasks concurrently in a large cluster ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[oilbeater]

This is related to #4822 , and #4835 remove some duplicate policy add. Can you check the logs to see what else happened between kube-ovn-controller restart and the controller start to process the queued event.

[zsxsoft]

Author of #4822, I experienced a similar issue with ~300 subnets and 10 nodes, where the kube-ovn-controller takes over 30 minutes to start.
The trend of AddSubnet event in my WFD chart is similar to your AddNode. I suspect that your subnets are centralized, as distributed subnets might take even longer to process.

[cmdy]

This is related to #4822 , and #4835 remove some duplicate policy add. Can you check the logs to see what else happened between kube-ovn-controller restart and the controller start to process the queued event.

this problem is caused by the execution of the init operation after the controller is started. In the init operation, initNodeRouter is very slow. I think the reason for the slowness is that there are too many addPolicyRouteToVpc operations. for this kind of initialization, batch operations should be used to reduce the number of accesses to the northbound database. i will submit a PR later, you can review it.

[cmdy]

PR is ready to be reviewed, please review @oilbeater
#4884