Aggregate NodeConfig status conditions

[rzetelskik]

Description of your changes: Currently, node setup controller only sets up conditions in the form of Node%sAvailable and the like. They are not practical and can't be easily used to query the NodeConfig status. This PR makes it so that NodeConfig conditions are aggregated to the generic Available, Progressing and Degraded, by:

• Replacing the proprietary NodeConfigCondition with metav1.Condition in NodeConfig API. What follows is that the helpers for NodeConfigCondition are no longer needed, so any related code is removed. Edit: we can't do in this API version due to metav1.Condition having a tighter validation. Left TODOs about making this change in the next API version.
• Adding conditions for NodeConfig controller
• Extending NodeConfig controller with conditions aggregation. Node setup controller still aggregates conditions per-node, but NodeConfig controller performs the top level aggregation. In case any of the nodes' aggregated conditions are missing, it assumes Available=False and Progressing=True for the given node.
• Deprecating the existing condition of type NodeConfigReconciledConditionType. The standard workload conditions are now also used to calculate the deprecated condition.
• Reacting to changes in existing E2E tests.

Which issue is resolved by this Pull Request:
Resolves #1557

/kind feature
/priority important-soon

/cc

[scylla-operator-bot]

@rzetelskik: GitHub didn't allow me to request PR reviews from the following users: rzetelskik.

Note that only scylladb members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

Description of your changes: WIP

Which issue is resolved by this Pull Request:
Resolves #1557

/kind feature
/priority important-soon

/cc

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[scylla-operator-bot]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[rzetelskik]

/test all

[rzetelskik]

/test images e2e-gke-serial

[rzetelskik]

/test all

[rzetelskik]

@tnozicka you mention here #1557 (comment) that the tests should be extended with trying out broken raid/mount configurations, but after thinking this through I don't think they should be a part of the e2e suite - condition aggregation is covered by unit tests, and invalid configurations shouldn't go past validation. Any scenarios with corrupted devices require some hacks on the host, are prone to be flaky and are difficult to even come up with - e.g. a scenario working on my local setup wouldn't reproduce in CI.

[rzetelskik]

/cc zimnx

[tnozicka]

invalid configurations shouldn't go past validation

I don't think validation has a chance to assess this. Say something is already mounted at the target path - it's not something to be assessed on the API level.

Any scenarios with corrupted devices require some hacks on the host, are prone to be flaky and are difficult to even come up with - e.g. a scenario working on my local setup wouldn't reproduce in CI.

I think it should be possible to come up with at least one without node changes, like trying to mount something over /dev/zero or similar but I don't feel strong.

[rzetelskik]

rebased and applied changes, @zimnx @tnozicka ptal

[rzetelskik]

/hold cancel

[rzetelskik]

rebased

[scylla-operator-bot]

@rzetelskik: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gke-release-script-latest	a20b428	link	true	/test e2e-gke-release-script-latest

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[rzetelskik]

@rzetelskik: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gke-release-script-latest a20b428 link true /test e2e-gke-release-script-latest
ci/prow/e2e-gke-parallel 77c0be6 link true /test e2e-gke-parallel
Full PR test history. Your PR dashboard.

known manager flake
#2061 (comment)

[tnozicka]

conditions are not technically part of the API version / can be unknown or unset - but I agree, it should be kept, at least for a while, to retain compatibility and behaviour

[tnozicka]

I'd put this into syncDaemonSet though, something like

	desiredSum := int64(0)
	allReconciled := true
	for _, requiredDaemonSet := range requiredDaemonSets {
		if requiredDaemonSet == nil {
			continue
		}

		ds, _, err := resourceapply.ApplyDaemonSet(ctx, ncc.kubeClient.AppsV1(), ncc.daemonSetLister, ncc.eventRecorder, requiredDaemonSet, resourceapply.ApplyOptions{})
		if err != nil {
			return progressingConditions, fmt.Errorf("can't apply daemonset: %w", err)
		}

		desiredSum += int64(ds.Status.DesiredNumberScheduled)

		reconciled, err := controllerhelpers.IsDaemonSetRolledOut(ds)
		if err != nil {
			return nil, fmt.Errorf("can't determine is a daemonset %q is reconiled: %w", naming.ObjRef(ds), err)
		}
		if !reconciled {
			allReconciled = false
		}
	}

	status.DesiredNodeSetupCount = pointer.Ptr(desiredSum)

	reconciledCondition := metav1.Condition{
		Type:               string(scyllav1alpha1.NodeConfigReconciledConditionType),
		ObservedGeneration: nc.Generation,
		Status:             metav1.ConditionUnknown,
	}

	if allReconciled {
		reconciledCondition.Status = metav1.ConditionTrue
		reconciledCondition.Reason = "FullyReconciledAndUp"
		reconciledCondition.Message = "All operands are reconciled and available."
	} else {
		reconciledCondition.Status = metav1.ConditionFalse
		reconciledCondition.Reason = "DaemonSetNotRolledOut"
		reconciledCondition.Message = "DaemonSet isn't reconciled and fully rolled out yet."
	}
	_ = apimeta.SetStatusCondition(statusConditions, reconciledCondition)

[rzetelskik]

@zimnx ping

[zimnx]

lgtm
/assign tnozicka

[tnozicka]

/approve
/lgtm

[tnozicka]

I haven't seen the new projection at a time I wrote this - looking at it, it changes the semantics of it but I guess that's ok

[scylla-operator-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rzetelskik, tnozicka, zimnx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [tnozicka,zimnx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment