-
Notifications
You must be signed in to change notification settings - Fork 5.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize processing managed resources in getResourceTree, ideally from quadratic to linear #18929
Comments
I'll try to implement this. |
|
Here's a |
…proj#18929) NOTE: this PR right now refences a gitops-engine in a fork for testing purposes Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Also, use more lenient locking when processing events (part of gitops-engine changes). Signed-off-by: Andrii Korotkov <[email protected]>
…proj#18929) NOTE: this PR right now refences a gitops-engine in a fork for testing purposes Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Also, use more lenient locking when processing events (part of gitops-engine changes). Signed-off-by: Andrii Korotkov <[email protected]>
…proj#18929) NOTE: this PR right now refences a gitops-engine in a fork for testing purposes Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Also, use more lenient locking when processing events (part of gitops-engine changes). Signed-off-by: Andrii Korotkov <[email protected]>
…proj#18929) NOTE: this PR right now refences a gitops-engine in a fork for testing purposes Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Also, use more lenient locking when processing events (part of gitops-engine changes). Signed-off-by: Andrii Korotkov <[email protected]>
…proj#18929) NOTE: this PR right now refences a gitops-engine in a fork for testing purposes Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Also, use more lenient locking when processing events (part of gitops-engine changes). Signed-off-by: Andrii Korotkov <[email protected]>
…proj#18929) NOTE: this PR right now refences a gitops-engine in a fork for testing purposes Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Also, use more lenient locking when processing events (part of gitops-engine changes). Signed-off-by: Andrii Korotkov <[email protected]>
Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Signed-off-by: Andrii Korotkov <[email protected]>
…oj#18972) Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Signed-off-by: Andrii Korotkov <[email protected]>
…oj#18972) Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Signed-off-by: Andrii Korotkov <[email protected]> Signed-off-by: Vegard Færgestad <[email protected]>
…oj#18972) Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Signed-off-by: Andrii Korotkov <[email protected]>
…oj#18972) Closes argoproj#18929 Helps with argoproj#18500 Use iterate hierarchy v2 to have a roughly linear performance for getting the resource tree instead of up to quadratic. Signed-off-by: Andrii Korotkov <[email protected]>
Summary
getResourceTree
has a loop through each of managed resources, callingIterateHierarchy
on state cache to go over children, which ends up callingIterateHierarchy
ingitops-engine
, which has a loop over resources in a namespace (https://github.com/argoproj/gitops-engine/blob/master/pkg/cache/cluster.go#L973), which callsiterateChildren
which has a similar loop (https://github.com/argoproj/gitops-engine/blob/master/pkg/cache/resource.go#L89). This is presumably since we only keep one-way track of child -> parent relationship, while effective traversal requires the opposite. Overall, this can result in a quadratic execution time ofO(tree_size * namespace_resources_count)
. If we pre-construct the graph with parent -> child edges, this can be reduced to linearO(namespace_resources_count)
.Motivation
getResourceTree
seems to be the slowest part of reconciliation for large apps, e.g. see timing data from a build with #18926:Getting resource tree took almost 4 minutes and was where almost all the reconciliation time went. It's an app with ~500 resources which don't even have children. And it's not the biggest cluster (staging). The biggest cluster (one of prod) can take 30-60 min to reconcile that app even with no changes.
The algorithm doesn't have to be quadratic and can be improved to linear.
Proposal
Pre-construct a graph from namespace resources parent -> child and do a linear dfs from each of managed resources, avoiding visiting same vertex twice.
gitops-engine
changes may not even be needed, but may be good to have anyways. Overall, this would reduce the time complexity from quadratic to linear.A sub-optimal fix can be to configure some resource groups and kinds as not having any children, so hierarchy iteration can be skipped for them.
The text was updated successfully, but these errors were encountered: