Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid coupling tests with hardcoded test CA certs #6086

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

y1hao
Copy link
Contributor

@y1hao y1hao commented Jan 25, 2025

What type of PR is this?

/kind flake

What this PR does / why we need it:

As we discovered in #6066, 2 test cases in membercluster_client_test.go depend directly on a hardcoded test CA certificate, which was duplicated from Go's net/http/internal package. This can cause these tests to fail when a new version of Go has regenerated that internal certificate. The test server constructed with Go's httptest package already exposes a method to retrieve its trusted CA. This PR updates the tests to use that method to retrieve the CA cert directly from the test server.

Other test cases in this file don't go through a test server, so they can use any cert. They don't need to be updated.

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

@karmada-bot karmada-bot added the kind/flake Categorizes issue or PR as related to a flaky test. label Jan 25, 2025
@karmada-bot karmada-bot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 25, 2025
@codecov-commenter
Copy link

codecov-commenter commented Jan 25, 2025

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 48.11%. Comparing base (d24b2b9) to head (6659377).
Report is 10 commits behind head on master.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #6086      +/-   ##
==========================================
- Coverage   48.35%   48.11%   -0.25%     
==========================================
  Files         666      668       +2     
  Lines       54880    55163     +283     
==========================================
+ Hits        26537    26541       +4     
- Misses      26618    26896     +278     
- Partials     1725     1726       +1     
Flag Coverage Δ
unittests 48.11% <ø> (-0.25%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Member

@RainbowMango RainbowMango left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/assign

@RainbowMango
Copy link
Member

Thank you @y1hao ! Excellent!

Other test cases in this file don't go through a test server, so they can use any cert. They don't need to be updated.

Yes! You are right, I tested it on my side and all tests passed after reverting the copied certificate.

Shall we deprecate (remove) the copied certificate? I think the right practice is like what you did in this PR, which generates a fresh certificate from the test server. Even the rest tests don't go through the test server, but they need a cert, why not give them a fresh one?

My concern about keeping the copied certificate is that sometimes the developers imitate the legacy tests when they want to introduce new test cases. They may meet the same issue again.

@karmada-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from rainbowmango. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@karmada-bot karmada-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 27, 2025
@y1hao
Copy link
Contributor Author

y1hao commented Jan 27, 2025

@RainbowMango Yeah, I agree that the best practice would be to use a fresh cert for each test case. But the concern here is that generating a new certificate is a heavy process, both in terms of compute resource consumed and the amount of code needed. For this reason, I think people usually don't create a new cert on the fly for testing, instead, they just use a pre-generated one.

When we retrieve the cert from the test server, it looks like that we are getting a fresh one for each test case, but in fact, we are just doing some decoding to retrieve the exact same cert every time, which is the previously copied one. If you compare the cert retrieved from the test server and the cert we copied from Go's net/http/internal, you'll find they are exactly the same. Go's test server doesn't bother to regenerate a new cert every time.

Similarly, k8s also has a pre-generated test certificate to use in tests: https://github.com/kubernetes/kubernetes/blob/f015fd66ce95d02cd66efc263eb9e5441b42a17d/hack/testdata/ca/ca.crt

So, I've updated this PR to use a new pre-generated certificate to replace the copied one, and documented the command used to generate it. This is the same practice as used in Go and k/k.

My concern about keeping the copied certificate is that sometimes the developers imitate the legacy tests when they want to introduce new test cases. They may meet the same issue again.

With the new cert, if someone incorrectly used the pre-generated test cert when they should retrieve the cert from the test server, their test will fail so they can fix this before merging. I've also moved the code to retrieve the cert from the test server to a helper method, to help people understand its purpose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants