Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Flake e2e (0)] Fail Forward Upgrades when a CSV resource is in a failed state [BeforeEach] eventually reports a successful state when using skips #3126

Closed
tmshort opened this issue Nov 30, 2023 · 1 comment
Closed

[Flake e2e (0)] Fail Forward Upgrades when a CSV resource is in a failed state [BeforeEach] eventually reports a successful state when using skips #3126

tmshort opened this issue Nov 30, 2023 · 1 comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test.

Comments

@tmshort
Copy link
Contributor

tmshort commented Nov 30, 2023

Flaky Test Report

Failure Log Link
Failure Log
You can get the link by clicking on the line number in the job logs and copying the url from your browser.

Relevant Failure Log

Fail Forward Upgrades when a CSV resource is in a failed state [BeforeEach] eventually reports a successful state when using skips
  [BeforeEach] /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:332
  [It] /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:403

  Captured StdOut/StdErr Output >>
  waiting 102.3187ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - NO_CONNECTION
  waiting 2.161734782s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - IDLE
  waiting 399.999906ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
  waiting 19.399318051s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
  waiting 999.950818ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
  waiting 20.000381764s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
  waiting 1.799416334s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - READY
  probing catalog mc-csv-failed-vpd7n pod with address mc-csv-failed-vpd7n.fail-forward-e2e-bhsmz.svc:50051
  skipping health check
  waiting 102.482175ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - NO_CONNECTION
  waiting 3.08146498s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - IDLE
  waiting 1.000002758s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
  waiting 18.002226617s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
  waiting 998.20021ms for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - CONNECTING
  waiting 20.00012404s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - TRANSIENT_FAILURE
  waiting 1.800247429s for catalog pod fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n to be available (for sync) - READY
  probing catalog mc-csv-failed-vpd7n pod with address mc-csv-failed-vpd7n.fail-forward-e2e-bhsmz.svc:50051
  skipping health check
  Using the kubectl kubectl binary
  Using the artifacts/fail-forward-e2e-bhsmz output directory
  Storing the test artifact output in the artifacts/fail-forward-e2e-bhsmz directory
  Collecting get catalogsources -o yaml output...
  Collecting get subscriptions -o yaml output...
  Collecting get operatorgroups -o yaml output...
  Collecting get clusterserviceversions -o yaml output...
  Collecting get installplans -o yaml output...
  Collecting get pods -o wide output...
  Collecting get events --sort-by .lastTimestamp output...
  << Captured StdOut/StdErr Output

  Timeline >>
  STEP: creating the testing namespace with an OG that enabled fail forward behavior @ 11/30/23 15:10:54.178
  created the fail-forward-e2e-bhsmz testing namespace
  STEP: deploying the testing catalog @ 11/30/23 15:10:54.193
  Creating catalog source mc-csv-failed-vpd7n in namespace fail-forward-e2e-bhsmz...
  Catalog source mc-csv-failed-vpd7n created
  STEP: creating the testing subscription @ 11/30/23 15:11:39.065
  STEP: waiting until the subscription has an IP reference @ 11/30/23 15:11:39.069
  15:11:41.8669: subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub state: UpgradePending (csv example-operator.v0.1.0): installPlanRef: &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-bhsmz", Name:"install-7b6j8", UID:"0f9b478e-f55c-4354-b374-b440b4b035ce", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"6838", FieldPath:""}
  waiting 2.797254321s for subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub to have installplan ref: has ref &v1.ObjectReference{Kind:"InstallPlan", Namespace:"fail-forward-e2e-bhsmz", Name:"install-7b6j8", UID:"0f9b478e-f55c-4354-b374-b440b4b035ce", APIVersion:"operators.coreos.com/v1alpha1", ResourceVersion:"6838", FieldPath:""}
  STEP: waiting for the v0.1.0 CSV to report a succeeded phase @ 11/30/23 15:11:41.867
  waiting for CSV fail-forward-e2e-bhsmz/example-operator.v0.1.0 to reach condition
  waited 199.903672ms for csv fail-forward-e2e-bhsmz/example-operator.v0.1.0 - Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
  waited 199.91354ms for CSV fail-forward-e2e-bhsmz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Installing (InstallWaiting): installing: waiting for deployment example-operator-controller-manager to become ready: deployment "example-operator-controller-manager" not available: Deployment does not have minimum availability.
  waited 398.714801ms for csv fail-forward-e2e-bhsmz/example-operator.v0.1.0 - Succeeded (InstallSucceeded): install strategy completed with no errors
  waited 398.716824ms for CSV fail-forward-e2e-bhsmz/example-operator.v0.1.0: to be in phases [Succeeded], in phase Succeeded (InstallSucceeded): install strategy completed with no errors
  STEP: updating the catalog with a broken v0.2.0 csv @ 11/30/23 15:11:42.465
  STEP: removing the existing catalog source @ 11/30/23 15:11:42.465
  STEP: removing the previous catalog source pod(s) @ 11/30/23 15:11:42.667
  waiting for the catalog source mc-csv-failed-vpd7n-s8rpm pod to be deleted...
  STEP: updating the catalog with a new bundle images @ 11/30/23 15:11:43.673
  Creating catalog source mc-csv-failed-vpd7n in namespace fail-forward-e2e-bhsmz...
  Catalog source mc-csv-failed-vpd7n created
  STEP: verifying the subscription has installed the current csv @ 11/30/23 15:12:28.666
  15:17:29.0655: subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub never got correct status: v1alpha1.SubscriptionStatus{CurrentCSV:"", InstalledCSV:"", Install:(*v1alpha1.InstallPlanReference)(nil), State:"", Reason:"", InstallPlanGeneration:0, InstallPlanRef:(*v1.ObjectReference)(nil), CatalogHealth:[]v1alpha1.SubscriptionCatalogHealth{v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc0007129a0), LastUpdated:time.Date(2023, time.November, 30, 15, 12, 40, 0, time.Local), Healthy:true}, v1alpha1.SubscriptionCatalogHealth{CatalogSourceRef:(*v1.ObjectReference)(0xc000712a10), LastUpdated:time.Date(2023, time.November, 30, 15, 12, 40, 0, time.Local), Healthy:true}}, Conditions:[]v1alpha1.SubscriptionCondition{v1alpha1.SubscriptionCondition{Type:"CatalogSourcesUnhealthy", Status:"False", Reason:"AllCatalogSourcesHealthy", Message:"all available catalogsources are healthy", LastHeartbeatTime:<nil>, LastTransitionTime:time.Date(2023, time.November, 30, 15, 12, 40, 0, time.Local)}, v1alpha1.SubscriptionCondition{Type:"ResolutionFailed", Status:"True", Reason:"ConstraintsNotSatisfiable", Message:"constraints not satisfiable: @existing/fail-forward-e2e-bhsmz//example-operator.v0.1.0, mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.2.0 and mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.1.0 originate from package packageA, clusterserviceversion example-operator.v0.1.0 exists and is not referenced by a subscription, subscription mc-csv-failed-vpd7n-sub requires at least one of mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.2.0 or mc-csv-failed-vpd7n/fail-forward-e2e-bhsmz/stable/example-operator.v0.1.0, subscription mc-csv-failed-vpd7n-sub exists", LastHeartbeatTime:<nil>, LastTransitionTime:<nil>}}, LastUpdated:time.Date(2023, time.November, 30, 15, 15, 56, 0, time.Local)}
  15:17:29.0655: subscription fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub spec: &v1alpha1.SubscriptionSpec{CatalogSource:"mc-csv-failed-vpd7n", CatalogSourceNamespace:"fail-forward-e2e-bhsmz", Package:"packageA", Channel:"stable", StartingCSV:"", InstallPlanApproval:"", Config:(*v1alpha1.SubscriptionConfig)(nil)}
  [FAILED] in [BeforeEach] - /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:374 @ 11/30/23 15:17:29.065
  STEP: removing the testing catalog resources @ 11/30/23 15:17:29.065
  Deleting config map mc-csv-failed-vpd7n-configmap...
  Deleting catalog source mc-csv-failed-vpd7n...
  waiting for the catalog source mc-csv-failed-vpd7n-tw2dk pod to be deleted...
  Deleting config map mc-csv-failed-vpd7n-configmap...
  Deleting catalog source mc-csv-failed-vpd7n...
  STEP: deleting the testing namespace @ 11/30/23 15:17:30.28
  collecting the fail-forward-e2e-bhsmz namespace artifacts as the 'eventually reports a successful state when using skips' test case failed
  collecting logs in the ./artifacts/ artifacts directory
  tearing down the fail-forward-e2e-bhsmz namespace
  resetting e2e kube client
  deleting fail-forward-e2e-bhsmz/mc-csv-failed-vpd7n-sub
  deleting fail-forward-e2e-bhsmz/fail-forward-e2e-bhsmz-operatorgroup
  deleting <global>/fail-forward-e2e-bhsmz
  garbage collecting CRDs
  << Timeline

  [FAILED] Expected
      <wait.errInterrupted>: 
      timed out waiting for the condition
      {
          cause: <*errors.errorString | 0xc000138140>{
              s: "timed out waiting for the condition",
          },
      }
  to be nil
  In [BeforeEach] at: /home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:374 @ 11/30/23 15:17:29.065

  Full Stack Trace
    github.com/operator-framework/operator-lifecycle-manager/test/e2e.glob..func14.4.1()
    	/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager/test/e2e/fail_forward_e2e_test.go:374 +0x827
@tmshort tmshort added the kind/flake Categorizes issue or PR as related to a flaky test. label Nov 30, 2023
@tmshort
Copy link
Contributor Author

tmshort commented Nov 19, 2024

Closing after a year. Our flakes have much improved.

@tmshort tmshort closed this as completed Nov 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/flake Categorizes issue or PR as related to a flaky test.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tmshort