Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

VPA recommender doesn't recommend CPU requests below 10m #6415

Open
dmitrii-sisutech opened this issue Jan 2, 2024 · 6 comments · May be fixed by #7554
Open

VPA recommender doesn't recommend CPU requests below 10m #6415

dmitrii-sisutech opened this issue Jan 2, 2024 · 6 comments · May be fixed by #7554
Labels
area/vertical-pod-autoscaler kind/feature Categorizes issue or PR as related to a new feature. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@dmitrii-sisutech
Copy link

Which component are you using?:
vertical-pod-autoscaler

What version of the component are you using?:

Component version: 1.0.0

What k8s version are you using (kubectl version)?:

kubectl version Output
$ kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.7-gke.1056000

What environment is this in?:
GKE

What did you expect to happen?:
I expect Recommender to provide CPU recommendations to VPAs less than 10m when consumption is less than 10m.

What happened instead?:
Instead the smallest recommended CPU is 10m while usage is 2-3m. On the screenshot there're 2 pods:
image

How to reproduce it (as minimally and precisely as possible):
If it's on GKE then if VPA is turned on turn it off in console.
Then

./hack/vpa-down.sh
./hack/vpa-up.sh

Anything else we need to know?:

Recommender configuration:

        - name: recommender
          image: registry.k8s.io/autoscaling/vpa-recommender:1.0.0
          imagePullPolicy: Always
          args:
            - --v=4
            - --cpu-histogram-decay-half-life=0h2m0s
            - --history-length="3h"
            - --history-resolution="5m"
            - --pod-recommendation-min-cpu-millicores=1

I tried also --pod-recommendation-min-cpu-millicores=5 and --pod-recommendation-min-cpu-millicores=1.0 - didn't change anything. To make sure that parameter works at all tried --pod-recommendation-min-cpu-millicores=50 and it worked.

While we stayed with GKE VPA it was setting up CPU recommendations down to 1m.
@dmitrii-sisutech dmitrii-sisutech added the kind/bug Categorizes issue or PR as related to a bug. label Jan 2, 2024
@dmitrii-sisutech
Copy link
Author

I think the problem is in that line.
The smallest bucket size is 0.01 that's assigned to firstBucketSize parameter here. Later that parameter is used in determining percentile here.

I tried setting it to 0.001 and ran it in my cluster and it worked. The least it was assigning is 2m and max that I've tried is ~700m.

@voelzmo
Copy link
Contributor

voelzmo commented Jan 11, 2024

Hey @dmitrii-sisutech,
Your observation is correct, the recommender's histogram doesn't have the resolution for usages <10m. Thanks for the comparison with the GKE VPA, I wasn't aware of this difference!

I'm happy to mark this issue as a feature request. If you want to file a PR for this, I can take a look an review it!

/remove-label bug
/kind feature

@k8s-ci-robot k8s-ci-robot added the kind/feature Categorizes issue or PR as related to a new feature. label Jan 11, 2024
@k8s-ci-robot
Copy link
Contributor

@voelzmo: The label(s) /remove-label bug cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to this:

Hey @dmitrii-sisutech,
Your observation is correct, the recommender's histogram doesn't have the resolution for usages <10m. Thanks for the comparison with the GKE VPA, I wasn't aware of this difference!

I'm happy to mark this issue as a feature request. If you want to file a PR for this, I can take a look an review it!

/remove-label bug
/kind feature

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@voelzmo
Copy link
Contributor

voelzmo commented Jan 11, 2024

/remove-kind bug

@k8s-ci-robot k8s-ci-robot removed the kind/bug Categorizes issue or PR as related to a bug. label Jan 11, 2024
@dmitrii-sisutech dmitrii-sisutech mentioned this issue Jan 11, 2024
Closed
@Shubham82
Copy link
Contributor

/triage accepted

@k8s-ci-robot k8s-ci-robot added the triage/accepted Indicates an issue or PR is ready to be actively worked on. label Jan 12, 2024
@adrianmoisey
Copy link
Member

/area vertical-pod-autoscaler

@glemsom glemsom linked a pull request Dec 3, 2024 that will close this issue
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/vertical-pod-autoscaler kind/feature Categorizes issue or PR as related to a new feature. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Milestone
No milestone
5 participants
@adrianmoisey @voelzmo @k8s-ci-robot @Shubham82 @dmitrii-sisutech