ServerSide Diff failing for CRs when a new field is added to the CRD: field not declared in schema

[varmarohit75]

Checklist:

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
  SImilar to Error: topologySpreadConstraints .matchLabelKeys: field not declared in schema #15176 but for CRD caching
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

We are using ArgoCD v2.10.9 to manage several CRDs deployed in our Kubernetes cluster.
Additionally, we have other applications that deploy CRs based on these CRDs through ArgoCD as well.

Whenever a new field is added to an existing CRD, it is successfully applied to the cluster, and all existing CRs are reconciled without issues. However, we encounter the following problem when trying to sync a CR based off this CRD through ArgoCD:

• Error on ArgoCD Sync:
  During the serverside diff operation, we receive the following error:

ComparisonError: Failed to compare desired state to live state: failed to calculate diff from cache: error calculating server side diff:
serverSideDiff error: error removing non config mutations for resource CustomStatefulSet/test-app-1:
error converting predicted live state from unstructured to example.com/v1alpha1, Kind=CustomStatefulSet: .spec.application.resourceRequirements: field not declared in schema

• Similar error is seen on Hard Refresh and sync:

ComparisonError:  Failed to compare desired state to live state: failed to calculate diff: error calculating server side diff:
serverSideDiff error: error removing non config mutations for resource CustomStatefulSet/test-app-1:
error converting predicted live state from unstructured to example.com/v1alpha1, Kind=CustomStatefulSet: .spec.application.resourceRequirements: field not declared in schema

To Reproduce

• Apply a CRD through ArgoCD with ServerSide diff set to true.
• Apply corresponding CRs for that CRD using ArgoCD.
• Add a new field to the CRD (preferably with a default value).
• Ensure that the new field is updated across all CRs in the Kubernetes cluster.
• Sync the CR in ArgoCD—this is where the error should occur.

Expected behavior

The cache to be reset timely when the updated CRD is applied.
Or when a sync is performed on the CR

Version

argocd: v2.11.3+3f344d5
  BuildDate: 2024-06-06T12:31:55Z
  GitCommit: 3f344d54a4e0bbbb4313e1c19cfe1e544b162598
  GitTreeState: clean
  GoVersion: go1.22.4
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.10.9+c071af8
  BuildDate: 2024-04-30T15:53:28Z
  GitCommit: c071af808170bfc39cbdf6b9be4d0212dd66db0c
  GitTreeState: clean
  GoVersion: go1.21.3
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.3+gf03cc04
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0
 

Logs

msg="Applying resource CustomStatefulSet/erbium-seas-cloud-searcher-i002 in cluster: https://10.244.0.1:443, namespace: seas-cloud-searcher-6a8f2" dry-run=server manager=argocd-controller serverSideApply=true serverSideDiff=true

msg="Normalized app spec: {\"status\":{\"conditions\":[{\"lastTransitionTime\":\"2024-10-17T19:40:50Z\",\"message\":\"Failed to compare desired state to live state: failed to calculate diff from cache: error calculating server side diff: serverSideDiff error: error removing non config mutations for resource CustomStatefulSet/erbium-seas-cloud-searcher-i002: error converting predicted live state from unstructured to example.com/v1alpha1, Kind=CustomStatefulSet: .spec.application.resourceRequirements: field not declared in schema\",\"type\":\"ComparisonError\"}]}}" application=argocd/seas-cloud-searcher

Mitigated through

A rollout restart of statefulset seems to fix the issue and the sync on CR goes through

kubectl rollout restart statefulset argocd-application-controller -n argocd