Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flake - ScyllaCluster [It] should re-bootstrap from old PVCs #2127

Open
zimnx opened this issue Sep 23, 2024 · 1 comment
Open

Flake - ScyllaCluster [It] should re-bootstrap from old PVCs #2127

zimnx opened this issue Sep 23, 2024 · 1 comment
Labels
kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.

Comments

@zimnx
Copy link
Collaborator

zimnx commented Sep 23, 2024

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/logs/ci-scylla-operator-enterprise-latest-e2e-gke-arm64-parallel/1838096006806245376

Snippet of what failed.

 ScyllaCluster [It] should re-bootstrap from old PVCs
github.com/scylladb/[email protected]/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:25
  Timeline >>
  STEP: Creating a new namespace @ 09/23/24 06:12:16.787
  Sep 23 06:12:16.807: INFO: Created namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:12:16.887
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:12:16.954
  STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:12:17.007
  STEP: Creating a ScyllaCluster @ 09/23/24 06:12:17.073
  STEP: Waiting for the ScyllaCluster to roll out (RV=5815) @ 09/23/24 06:12:17.267
  [FAILED] in [It] - github.com/scylladb/[email protected]/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:44 @ 09/23/24 06:37:47.269
  STEP: Collecting events from namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:37:47.27
  STEP: Found 124 events. @ 09/23/24 06:37:47.288
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:17 +0000 UTC - event for basic-f4qbj-auth-token: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-auth-token created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:17 +0000 UTC - event for basic-f4qbj-member: {scyllacluster-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-member created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:17 +0000 UTC - event for basic-f4qbj-member: {scyllacluster-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-member created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:28 +0000 UTC - event for basic-f4qbj-local-client-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-client-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:12:28 +0000 UTC - event for basic-f4qbj-local-client-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-client-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:13:12 +0000 UTC - event for basic-f4qbj-local-user-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-user-admin created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:13:52 +0000 UTC - event for basic-f4qbj-local-serving-ca: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:13:52 +0000 UTC - event for basic-f4qbj-local-serving-ca: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-ca created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj: {scyllacluster-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj: {controllermanager } NoPods: No matching pods found
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-client: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-client created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-managed-config: {scyllacluster-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-managed-config created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-0 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-1 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:14:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceCreated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-2 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-local-cql-connection-configs-admin: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-cql-connection-configs-admin created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-local-serving-certs: {scyllacluster-controller } SecretCreated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-certs created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {scyllacluster-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-f4qbj-us-east-1-us-east-1a-0 in StatefulSet basic-f4qbj-us-east-1-us-east-1a successful
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-f4qbj-us-east-1-us-east-1a-0 Pod basic-f4qbj-us-east-1-us-east-1a-0 in StatefulSet basic-f4qbj-us-east-1-us-east-1a success
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } ProvisioningSucceeded: Successfully provisioned volume pvc-4520cd7b-0b87-4440-a7c1-a277b05b3c8a
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-tl9cs-0-4xm2h/data-basic-f4qbj-us-east-1-us-east-1a-0"
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:26 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:27 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-0 to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:27 +0000 UTC - event for nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:28 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:28 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:29 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:45 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:47 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.40:10001: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: Get "http://10.70.1.40:42081/readyz": dial tcp 10.70.1.40:42081: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.40:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:15:49 +0000 UTC - event for nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6d6db659-8557-46d9-99f9-2687c64aafb7 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:08 +0000 UTC - event for basic-f4qbj-local-serving-certs: {scyllacluster-controller } SecretUpdated: Secret e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-local-serving-certs updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-f4qbj-us-east-1-us-east-1a-1 in StatefulSet basic-f4qbj-us-east-1-us-east-1a successful
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-f4qbj-us-east-1-us-east-1a-1 Pod basic-f4qbj-us-east-1-us-east-1a-1 in StatefulSet basic-f4qbj-us-east-1-us-east-1a success
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-tl9cs-0-4xm2h/data-basic-f4qbj-us-east-1-us-east-1a-1"
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } ProvisioningSucceeded: Successfully provisioned volume pvc-fa60535d-cd68-430f-9ec5-7cf99c2f4ad6
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:38 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-1 to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:38 +0000 UTC - event for nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:41 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:48 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-0 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:16:59 +0000 UTC - event for nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-6e2c3736-f018-491b-941d-b9c42a3a0a86 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:00 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.51:10001: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:00 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.51:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:00 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: Get "http://10.70.1.51:42081/readyz": dial tcp 10.70.1.51:42081: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:08 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Startup probe failed: Get "http://10.70.1.51:8080/healthz": dial tcp 10.70.1.51:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-f4qbj-us-east-1-us-east-1a-2 Pod basic-f4qbj-us-east-1-us-east-1a-2 in StatefulSet basic-f4qbj-us-east-1-us-east-1a success
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-f4qbj-us-east-1-us-east-1a-2 in StatefulSet basic-f4qbj-us-east-1-us-east-1a successful
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:37 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-tl9cs-0-4xm2h/data-basic-f4qbj-us-east-1-us-east-1a-2"
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:38 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-2 to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:38 +0000 UTC - event for data-basic-f4qbj-us-east-1-us-east-1a-2: {local.csi.scylladb.com_local-csi-driver-k99cb_6533e7e4-60a5-4347-aefd-60a59b693fb9 } ProvisioningSucceeded: Successfully provisioned volume pvc-4ee228a5-62ef-4c88-bc26-0bbfd1338c46
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:38 +0000 UTC - event for nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:39 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:40 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container sidecar-injection
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:49 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-1 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:54 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:54 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:55 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:55 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:55 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-ignition
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Pulled: Container image "docker.io/scylladb/scylla-operator-enterprise@sha256:2cc8d49b8547586d4b708a7fae17289d26cc891ced3cc2e4deaefd8ca17ca7b9" already present on machine
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:56 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylladb-api-status-probe
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.59:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: Get "http://10.70.1.59:42081/readyz": dial tcp 10.70.1.59:42081: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Created: Created container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Started: Started container scylla-manager-agent
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:57 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Readiness probe failed: dial tcp 10.70.1.59:10001: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:58 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {kubelet gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd} Unhealthy: Startup probe failed: Get "http://10.70.1.59:8080/healthz": dial tcp 10.70.1.59:8080: connect: connection refused
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:17:58 +0000 UTC - event for nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-tl9cs-0-4xm2h/nodeconfig-podinfo-3ebb6dbe-7c71-4a0c-9193-2e5c88bc123c updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for basic-f4qbj-us-east-1-us-east-1a-2: {scyllacluster-controller } ServiceUpdated: Service e2e-test-scyllacluster-tl9cs-0-4xm2h/basic-f4qbj-us-east-1-us-east-1a-2 updated
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } JobCreated: Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0 created
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-f4qbj-us-east-1-us-east-1a-0-7ts8j
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:50 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0-7ts8j: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0-7ts8j to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-f4qbj-us-east-1-us-east-1a-0-5xrnb
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0: {scyllacluster-controller } JobDeleted: Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0 deleted
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-0-5xrnb: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-0-5xrnb to gke-so-ef7664e5-0168-495f-861-workers-60851e9f-3zkd
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-1: {job-controller } Completed: Job completed
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } JobDeleted: Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-1 deleted
  Sep 23 06:37:47.288: INFO: At 2024-09-23 06:18:56 +0000 UTC - event for cleanup-basic-f4qbj-us-east-1-us-east-1a-1: {scyllacluster-controller } DeleteJobFailed: Failed to delete Job e2e-test-scyllacluster-tl9cs-0-4xm2h/cleanup-basic-f4qbj-us-east-1-us-east-1a-1: jobs.batch "cleanup-basic-f4qbj-us-east-1-us-east-1a-1" not found
  STEP: Collecting dumps from namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:37:47.288
  STEP: Destroying namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h". @ 09/23/24 06:37:48.183
  STEP: Waiting for namespace "e2e-test-scyllacluster-tl9cs-0-4xm2h" to be removed. @ 09/23/24 06:37:48.194
  << Timeline
  [FAILED] Unexpected error:
      <*fmt.wrapError | 0x4000a200e0>: 
      waiting has been interupted ([false]): timed out waiting for the condition
      {
          msg: "waiting has been interupted ([false]): timed out waiting for the condition",
          err: <wait.errInterrupted>{
              cause: <*errors.errorString | 0x4000334890>{
                  s: "timed out waiting for the condition",
              },
          },
      }
  occurred
  In [It] at: github.com/scylladb/[email protected]/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:44 @ 09/23/24 06:37:47.269
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func10.1()
    	github.com/scylladb/[email protected]/test/e2e/set/scyllacluster/scyllacluster_rebootstrap.go:44 +0x344 
    	```
@zimnx zimnx added the kind/flake Categorizes issue or PR as related to a flaky test. label Sep 23, 2024
@scylla-operator-bot scylla-operator-bot bot added the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Sep 23, 2024
@tnozicka tnozicka added the priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. label Sep 24, 2024
@scylla-operator-bot scylla-operator-bot bot removed the needs-priority Indicates a PR lacks a `priority/foo` label and requires one. label Sep 24, 2024
@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out

/lifecycle stale

@scylla-operator-bot scylla-operator-bot bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/flake Categorizes issue or PR as related to a flaky test. lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@zimnx @tnozicka