Add e2e tests for ProviderConfigs #1320

erhancagirici · 2024-05-23T09:58:38Z

Description of your changes

Introduces automatede2e tests for provider configs,

provider-aws supports several provider config scenarios such as IRSA, WebIdentity. To test these scenarios, one needs to prepare an testing environment manually, and testing auth methods like IRSA is only possible with an EKS cluster .

The change aims to provide an automated standard testing environment in EKS and conduct e2e tests for several predetermined scenarios. It also provides a base for extending the tests.

For the testing environment and the tests a new configuration package is introduced specifically targeting e2e testing. It introduces a new composite resource xe2etestclusters.platformref.aws.upbound.io, that provisions an IRSA-enabled EKS, deploys crossplane, providers, provider configs, and some built-in testing MRs.

Currently, built-in scenarios include:

IRSA
IRSA with a chained role
Legacy WebIdentity
WebIdentity with token at filesystem
WebIdentity with token at filesystem + chained Role

Starting with a local crossplane control plane (possibly in a local KinD cluster), when a E2ETestCluster.platformref.aws.upbound.io claim is created, all above will be provisioned and tested. If further tests need to be conducted with different provider configs, one can keep the claim and apply desired provider config and MR manifests to the remote EKS testing cluster, via provider-kubernetes & produced k8s provider config .

See readme at path e2e/providerconfig-aws-e2e-test/README.md further details on Structure & Usage.

I have:

Read and followed Crossplane's contribution process.
Run make reviewable to ensure this PR is ready for review.
Added backport release-x.y labels to auto-backport this PR if necessary.

How has this code been tested

Tested manually and the newly introduced providerconfig-e2e make target.

Signed-off-by: Erhan Cagirici <[email protected]>

mbbush

I only made it partway through before having to pause my review. So far my comments are mostly just questions or optional suggestions.

mbbush · 2024-06-05T17:22:24Z

Makefile

+# It first builds and publishes the
+# provider-family-aws, provider-aws-ec2 and provider-aws-rds.
+# Then triggers the e2e provider config tests via `make`,
+# which resides in the `e2e/providerconfig-aws-e2e-test` directory


This seems like a name that's longer than necessary to convey the information. How about just e2e/providerconfig

I am fine with the renaming the folder. My reasoning was, since this directory is also an XP configuration package, I just named it the same with the configuration package name.

The reason for this long name for the configuration package is that we could have other testing configuration packages for other providers as well.

mbbush · 2024-06-05T17:41:34Z

Makefile

@@ -209,6 +209,38 @@ uptest: $(UPTEST) $(KUBECTL) $(KUTTL)
 	@KUBECTL=$(KUBECTL) KUTTL=$(KUTTL) CROSSPLANE_NAMESPACE=$(CROSSPLANE_NAMESPACE) $(UPTEST) e2e "${UPTEST_EXAMPLE_LIST}" --data-source="${UPTEST_DATASOURCE_PATH}" --setup-script=cluster/test/setup.sh --default-conditions="Test" || $(FAIL)
 	@$(OK) running automated tests

+# This target triggers an e2e test for testing provider configs.
+# It first builds and publishes the
+# provider-family-aws, provider-aws-ec2 and provider-aws-rds.


Choosing two providers plus the family makes sense, but ec2 is by far the largest provider with 100 managed resource kinds. We might get better performance with the same confidence of catching regressions using two smaller providers that have a cross reference. How about kms and secretsmanager?

In the context of these tests, we only aim to test different provider configs. Any type of MR, therefore provider, works. We are not particularly interested in the External Resource itself, we just want to create MRs pointing to different providerconfigs in test.

I have no strong opinion on the resources/providers used for testing.

The reason behind EC2 VPC and RDS ParameterGroup is that they have

relatively fast creation & readiness time,

can be created standalone without any extra AWS resource

does not incur extra costs during testing or if somehow they become orphan

The reason for having 2 providers is that, IRSA tests have injected identity in the pod, I just wanted to have an isolated environment for other tests in a separate provider pod that does not have any IRSA config involved.

mbbush · 2024-06-05T17:45:58Z

Makefile

+	AWS_FAMILY_PACKAGE_IMAGE="$(XPKG_REG_ORGS)/provider-family-aws:$(VERSION)" \
+	AWS_EC2_PACKAGE_IMAGE="$(XPKG_REG_ORGS)/provider-aws-ec2:$(VERSION)" \
+	AWS_RDS_PACKAGE_IMAGE="$(XPKG_REG_ORGS)/provider-aws-rds:$(VERSION)" \
+	TARGET_CROSSPLANE_VERSION="1.15.2" \


I'm surprised to see this, as I thought there was a difference in crossplane versions >= 1.15.0 related to how it identifies family providers that resulted in the providers we build locally being identified as being from different families, and breaking access to the ProviderConfig and ProviderConfigUsage resources.

Assuming your tests are working, it would be great to figure out what's different about them and the current make e2e config that causes make e2e to fail with crossplane versions >= 1.15.0.

mbbush · 2024-06-05T17:47:34Z

e2e/providerconfig-aws-e2e-test/Makefile

@@ -0,0 +1,106 @@
+# Project Setup
+PROJECT_NAME := providerconfig-aws-e2e-test
+PROJECT_REPO := github.com/upbound/provider-upjet-aws


This URL returns a 404. I thought it was either github.com/crossplane-contrib/provider-upjet-aws or github.com/upbound/provider-aws. But maybe we're using a different convention here?

It has been some time since this PR was initiated, so probably it switched orgs after I opened the PR. Will fix this

mbbush · 2024-06-05T17:49:01Z

e2e/providerconfig-aws-e2e-test/Makefile

+# ====================================================================================
+# Setup XPKG
+XPKG_DIR = $(shell pwd)/package
+XPKG_IGNORE = .github/workflows/*.yaml,.github/workflows/*.yml,examples/*.yaml,.work/uptest-datasource.yaml,.cache/**,_output/**


Any reason not to entirely exclude .github/**, .work/**, examples/** and examples-generated/**?

mbbush · 2024-06-05T17:50:07Z

e2e/providerconfig-aws-e2e-test/Makefile

+
+e2e-lite: build controlplane.up local.xpkg.deploy.configuration.$(PROJECT_NAME)
+
+uptest-e2e: $(UPTEST) $(KUBECTL) $(KUTTL) $(YQ)


These names are getting confusing. We already have make uptest and make e2e in the root makefile.

sergenyalcin

@erhancagirici LGTM. Thank you very much for your great effort in this PR and adding the very critical e2e tests for different type of provider configs.

sergenyalcin · 2024-08-07T14:36:38Z

e2e/providerconfig-aws-e2e-test/Makefile

+KIND_VERSION = v0.22.0
+UP_VERSION = v0.28.0
+UP_CHANNEL = stable
+UPTEST_VERSION = v0.11.1


Please upgrade to the latest one v0.13.1. Also, now we are using the differente source for downloading the uptest artifacts. Please see the UPTEST_LOCAL in the root Makefile.

sergenyalcin · 2024-08-07T14:39:11Z

e2e/providerconfig-aws-e2e-test/README.md

+```
+
+### Via Github Actions from PR 
+TBD


nit: Do you think address these TBDs on this PR or future?

sergenyalcin · 2024-08-07T14:49:47Z

e2e/providerconfig-aws-e2e-test/Makefile

+uptest-e2e: $(UPTEST) $(KUBECTL) $(KUTTL) $(YQ)
+	@$(INFO) dump e2e claim:
+	@mkdir -p "_output"
+	@$(YQ) '(.spec.parameters.targetClusterParameters.provider.familyPackage = env(AWS_FAMILY_PACKAGE_IMAGE)) | \


Adding YQ to the build submodule worked here too 🎉

Add e2e tests for ProviderConfigs

329f265

Signed-off-by: Erhan Cagirici <[email protected]>

erhancagirici marked this pull request as ready for review May 28, 2024 09:03

erhancagirici requested review from ulucinar, sergenyalcin, turkenf and mbbush as code owners May 28, 2024 09:03

mbbush reviewed Jun 5, 2024

View reviewed changes

sergenyalcin approved these changes Aug 7, 2024

View reviewed changes

erhancagirici mentioned this pull request Aug 13, 2024

feat(pod-identity): add option for pod-identity #1459

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e tests for ProviderConfigs #1320

Add e2e tests for ProviderConfigs #1320

erhancagirici commented May 23, 2024

mbbush left a comment

mbbush Jun 5, 2024

erhancagirici Aug 7, 2024

mbbush Jun 5, 2024

erhancagirici Aug 7, 2024

mbbush Jun 5, 2024

mbbush Jun 5, 2024

erhancagirici Aug 7, 2024

mbbush Jun 5, 2024

mbbush Jun 5, 2024

sergenyalcin left a comment

sergenyalcin Aug 7, 2024

sergenyalcin Aug 7, 2024

sergenyalcin Aug 7, 2024


		e2e-lite: build controlplane.up local.xpkg.deploy.configuration.$(PROJECT_NAME)

		uptest-e2e: $(UPTEST) $(KUBECTL) $(KUTTL) $(YQ)

Add e2e tests for ProviderConfigs #1320

Are you sure you want to change the base?

Add e2e tests for ProviderConfigs #1320

Conversation

erhancagirici commented May 23, 2024

Description of your changes

How has this code been tested

mbbush left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sergenyalcin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment