Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(resource): add resource limits cases #187

Open
wants to merge 20 commits into
base: main
Choose a base branch
from
Open

Conversation

jijiechen
Copy link
Member

@jijiechen jijiechen commented May 27, 2024

  1. Run the control plane and connect specified numbers of DPs onto it and assert it should not fail.
  2. Compare the runnings between settings of GOMEMLIMIT.
# set the amount of memory limit onto the CP workload in MiB
export LIMIT_MEGA_MEMORY=256
# set the amount of CPU limit onto the CP workload in millicores
export LIMIT_MILLI_CPU=1000

# set the numbers of services
export PERF_TEST_NUM_SERVICES=5
# set the numbers of instances per service
export PERF_TEST_INSTANCES_PER_SERVICE=20

make run

Cases included:

  1. should deploy all services and instances
  2. should crash when deploy all services and instances with half CP resource
  3. should not crash when control plane has GOMEMLIMIT set when all services and instances with half CP resource

A lot of metrics are periodically fetched and printed to output, so I'm trying to save it to a file, otherwise it'll be troublesome to open the view for the GHA runs.

fixes #235

@Icarus9913
Copy link
Member

Almost LGTM

test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
_, _ = fmt.Fprintf(GinkgoWriter, "Pod %s: %s\n", pod.Name, pod.Status.Phase)
for _, cts := range pod.Status.ContainerStatuses {
_, _ = fmt.Fprintf(GinkgoWriter, "Pod %s - %s: %s\n", pod.Name, cts.Name, cts.State.String())
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we printing this? Should this be assertion or what's the reason?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I want to print dp statuses to help diagnose issues. For the case when "control plane should not crash", the DPs are also expected to be healthy. So if there is any DP unready, it should be not related to the CP.

test/k8s/res_limits_test.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add perf test for GOMEMLIMIT env on the control plane
3 participants