[Bug]: cannot resolve package dependencies

[applike-ss]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

all?

Resource MRs required to reproduce the bug

none

Steps to Reproduce

Haven't checked if it is reproducable

What happened?

Since today I am seeing some provider revisions aren't healthy.

All of them are providers from this repo.

They do have a status message like:

cannot resolve package dependencies: incompatible dependencies: existing
      package xpkg.upbound.io/upbound/provider-family-aws@v1 is incompatible
      with constraint v1.23.2

It might've been already since the license issue (#1829) on friday, but I have to check.

Relevant Error Output Snippet

Crossplane Version

v1.20.0-up.1

Provider Version

v1

Kubernetes Version

1.33

Kubernetes Distribution

EKS

Additional Info

No response

[applike-ss]

I just checked the status' on multiple clusters and the earliest occurence is on 1st of August ~10:55UTC. That sounds suspiciously close to be related to #1829

We are currently trying out to set a fixed version v1.23.2 instead of v1 and that seems to work in first tests.

[HeinanCA]

Adding to that - for us, this began on 2nd of August ~18:00UTC when Crossplane tried to reconcile resources on its own (as designed). This caused 5 different clusters to go into "Unknown" state in ArgoCD - with the same error message as OP.

After manually setting the version, crossplane (v1.20) simply tried to re-create the resources - which caused the whole EKS cluster to stop responding (because VPC was gone). Eventually we had to re-create the entire enviroment from a backup.

We fail to see how such a minor change (v1.23.1 --> v1.23.2) gets such a huge change, without notifying customers, who use it on production. This is simply bad practice - such a major change should involve a major version number change.

[applike-ss]

For us setting it to v1.23.2 seems to have worked without any impact, then setting it back to v1 to ensure it works nicely with the next provider update.

As it seems: mileage may vary.

[HeinanCA]

This regression has been affecting multiple production clusters for two weeks now. Could we get an update on:

• Timeline for a fix/rollback
• Root cause analysis
• Why breaking changes were included in a patch release (violating semver)?

Many organisations are blocked on upgrades due to this issue. Any guidance on workarounds would also be appreciated.
Thanks!

[jorgelon]

Downgrading to v1.23.1 worked for me. Waiting for a fixed release. I hope the 1.23.1 can be available until v1.23.2 is fixed

[YuriiBudnyi]

Got the same issue in our clusters. Explicitly set v1.23.1 fixed an issue. Waiting for a fix

[YuriiBudnyi]

Resolved this issue by explicitly setting the provider upbound-provider-family-aws version to v1.23.1 -> v1.23.2 -> v1