Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flake - should setup and maintain up to date TLS certificates #2096

Open
zimnx opened this issue Aug 27, 2024 · 5 comments
Open

Flake - should setup and maintain up to date TLS certificates #2096

zimnx opened this issue Aug 27, 2024 · 5 comments
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@zimnx
Copy link
Collaborator

zimnx commented Aug 27, 2024

Link to the job that flaked.

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2066/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1828179211538403328

Snippet of what failed.

 ScyllaCluster [It] should setup and maintain up to date TLS certificates
github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:42
  Timeline >>
  STEP: Creating a new namespace @ 08/26/24 21:30:43.436
  Aug 26 21:30:43.466: INFO: Created namespace "e2e-test-scyllacluster-2ms98-0-85njt".
  STEP: Waiting for service account token Secret "e2e-user-token" in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:43.522
  STEP: Waiting for default ServiceAccount in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:44.178
  STEP: Waiting for kube-root-ca.crt in namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:30:44.492
  STEP: Creating an initial ScyllaCluster with a single node @ 08/26/24 21:30:44.502
  STEP: Waiting for the ScyllaCluster to roll out (RV=5639) @ 08/26/24 21:30:44.533
  Aug 26 21:35:13.491: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11632) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:13.491
  Aug 26 21:35:13.648: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:35:13.648: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:13.757
  Aug 26 21:35:14.163: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Inserting data @ 08/26/24 21:35:14.259
  Aug 26 21:35:14.259: INFO: Creating CQL session (hosts="10.17.250.51")
  STEP: Inserting data @ 08/26/24 21:35:14.263
  Aug 26 21:35:14.263: INFO: Creating keyspace "6pztcv98" with RF "'replication_factor': 1"
  Aug 26 21:35:14.271: INFO: Creating table "6pztcv98"."test"
  Aug 26 21:35:14.281: INFO: Inserting data into table "6pztcv98"."test"
  Aug 26 21:35:14.296: INFO: Awaiting schema agreement
  Aug 26 21:35:14.297: INFO: Schema agreement reached
  STEP: Verifying the data @ 08/26/24 21:35:14.297
  Aug 26 21:35:14.297: INFO: Reading data from table "6pztcv98"."test"
  STEP: Scaling the ScyllaCluster to 1 replicas setting domains to [] @ 08/26/24 21:35:14.3
  STEP: Waiting for the ScyllaCluster to roll out (RV=11670) @ 08/26/24 21:35:14.496
  Aug 26 21:35:14.643: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11670) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:14.648
  Aug 26 21:35:14.905: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:35:14.905: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:14.981
  Aug 26 21:35:15.253: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Verifying TLS API objects @ 08/26/24 21:35:15.326
  STEP: Verifying certificates @ 08/26/24 21:35:15.384
  Aug 26 21:35:15.456: INFO: Starting to probe node "10.17.250.51" for correct certs
  Aug 26 21:35:15.530: INFO: Node "10.17.250.51" has correct certs
  STEP: Scaling the ScyllaCluster to 1 replicas setting domains to ["foo.scylladb.com" "bar.scylladb.com"] @ 08/26/24 21:35:15.53
  STEP: Waiting for the ScyllaCluster to roll out (RV=11738) @ 08/26/24 21:35:15.601
  Aug 26 21:35:16.039: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=11775) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:35:16.039
  Aug 26 21:35:16.296: INFO: Found 1 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:35:16.296: INFO: Found 1 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:35:16.458
  Aug 26 21:35:16.644: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Verifying TLS API objects @ 08/26/24 21:35:16.702
  STEP: Verifying certificates @ 08/26/24 21:35:16.856
  Aug 26 21:35:16.927: INFO: Starting to probe node "10.17.250.51" for correct certs
  Aug 26 21:36:11.572: INFO: Node "10.17.250.51" has correct certs
  STEP: Scaling the ScyllaCluster to 2 replicas setting domains to ["foo.scylladb.com" "bar.scylladb.com"] @ 08/26/24 21:36:11.572
  STEP: Waiting for the ScyllaCluster to roll out (RV=13780) @ 08/26/24 21:36:11.604
  Aug 26 21:37:44.388: INFO: ScyllaCluster e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 (RV=16384) is rolled out
  STEP: Verifying the ScyllaCluster @ 08/26/24 21:37:44.389
  Aug 26 21:37:44.431: INFO: Found 2 pvc(s) in namespace "e2e-test-scyllacluster-2ms98-0-85njt"
  Aug 26 21:37:44.431: INFO: Found 2 pvc(s) for ScyllaCluster "e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8"
  STEP: Waiting for the ScyllaCluster(s) to reach consistency ALL @ 08/26/24 21:37:44.496
  Aug 26 21:37:44.576: INFO: ScyllaDB nodes have reached status consistency.
  STEP: Verifying TLS API objects @ 08/26/24 21:37:44.608
  STEP: Verifying certificates @ 08/26/24 21:37:44.654
  [FAILED] in [It] - github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 @ 08/26/24 21:37:44.68
  STEP: Collecting events from namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:44.68
  STEP: Found 85 events. @ 08/26/24 21:37:44.694
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:48 +0000 UTC - event for basic-nztd8: {scyllaclustermigration-controller } ScyllaDBDatacenterCreated: ScyllaDBDatacenter e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:48 +0000 UTC - event for basic-nztd8-member: {scylladbdatacenter-controller } ServiceAccountCreated: ServiceAccount e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-member created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:49 +0000 UTC - event for basic-nztd8-member: {scylladbdatacenter-controller } RoleBindingCreated: RoleBinding e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-member created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:30:50 +0000 UTC - event for basic-nztd8-auth-token: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-auth-token created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:05 +0000 UTC - event for basic-nztd8-local-client-ca: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-client-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:05 +0000 UTC - event for basic-nztd8-local-client-ca: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-client-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:31:41 +0000 UTC - event for basic-nztd8-local-user-admin: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-user-admin created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:10 +0000 UTC - event for basic-nztd8-local-serving-ca: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:10 +0000 UTC - event for basic-nztd8-local-serving-ca: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-ca created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8: {scylladbdatacenter-controller } PodDisruptionBudgetCreated: PodDisruptionBudget e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8: {controllermanager } NoPods: No matching pods found
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-client: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-client created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-managed-config: {scylladbdatacenter-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-managed-config created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:32:43 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-local-cql-connection-configs-admin: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-cql-connection-configs-admin created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-local-serving-certs: {scylladbdatacenter-controller } SecretCreated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-certs created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {scylladbdatacenter-controller } StatefulSetCreated: StatefulSet e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-nztd8-us-east-1-us-east-1a-0 in StatefulSet basic-nztd8-us-east-1-us-east-1a successful
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-nztd8-us-east-1-us-east-1a-0 Pod basic-nztd8-us-east-1-us-east-1a-0 in StatefulSet basic-nztd8-us-east-1-us-east-1a success
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-2ms98-0-85njt/data-basic-nztd8-us-east-1-us-east-1a-0"
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:20 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-0: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } ProvisioningSucceeded: Successfully provisioned volume pvc-80a399a7-965f-4ccb-b26d-9a14946e88ea
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:21 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:21 +0000 UTC - event for nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:23 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:26 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:33:28 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:02 +0000 UTC - event for basic-nztd8-local-serving-certs: {scylladbdatacenter-controller } SecretUpdated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-serving-certs updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:16 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:19 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:19 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:23 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:25 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:28 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:32 +0000 UTC - event for nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-26139921-f3df-4a43-995e-b9f665409456 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.50:8080: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: Get "http://10.125.193.50:42081/readyz": dial tcp 10.125.193.50:42081: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:33 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.50:10001: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:41 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 503
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:41 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Startup probe failed: HTTP probe failed with statuscode: 503
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:34:52 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: HTTP probe failed with statuscode: 500
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:13 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } ServiceUpdated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-0 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:15 +0000 UTC - event for basic-nztd8: {scyllaclustermigration-controller } ScyllaDBDatacenterUpdated: ScyllaDBDatacenter e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:35:15 +0000 UTC - event for basic-nztd8-local-cql-connection-configs-admin: {scylladbdatacenter-controller } SecretUpdated: Secret e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-local-cql-connection-configs-admin updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:11 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {scylladbdatacenter-controller } ServiceCreated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Pod basic-nztd8-us-east-1-us-east-1a-1 in StatefulSet basic-nztd8-us-east-1-us-east-1a successful
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {statefulset-controller } SuccessfulCreate: create Claim data-basic-nztd8-us-east-1-us-east-1a-1 Pod basic-nztd8-us-east-1-us-east-1a-1 in StatefulSet basic-nztd8-us-east-1-us-east-1a success
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {persistentvolume-controller } WaitForFirstConsumer: waiting for first consumer to be created before binding
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } ProvisioningSucceeded: Successfully provisioned volume pvc-ebb938df-3693-4f56-adb3-fc8ef5f6cb72
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {persistentvolume-controller } ExternalProvisioning: Waiting for a volume to be created either by the external provisioner 'local.csi.scylladb.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:12 +0000 UTC - event for data-basic-nztd8-us-east-1-us-east-1a-1: {local.csi.scylladb.com_local-csi-driver-jrndw_b6271f4d-bd61-41ae-9a05-e05fa1264764 } Provisioning: External provisioner is provisioning volume for claim "e2e-test-scyllacluster-2ms98-0-85njt/data-basic-nztd8-us-east-1-us-east-1a-1"
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:13 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:13 +0000 UTC - event for nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f: {NodeConfigCM-controller } ConfigMapCreated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:15 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:17 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container sidecar-injection
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:43 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla:6.1.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:44 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:45 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:46 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "quay.io/scylladb-dev/ci:scylla-operator-pr-73b4d6614bf33d5d77ff213621289c2" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:46 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-api-status-probe
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:47 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylladb-ignition
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Started: Started container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Pulled: Container image "docker.io/scylladb/scylla-manager-agent:3.3.0" already present on machine
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:48 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Created: Created container scylla-manager-agent
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:49 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.92:8080: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:49 +0000 UTC - event for nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f: {NodeConfigCM-controller } ConfigMapUpdated: ConfigMap e2e-test-scyllacluster-2ms98-0-85njt/nodeconfig-podinfo-e3f5fefe-8131-43c6-872f-014e4043e82f updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:50 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: dial tcp 10.125.193.92:10001: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:36:50 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {kubelet gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl} Unhealthy: Readiness probe failed: Get "http://10.125.193.92:42081/readyz": dial tcp 10.125.193.92:42081: connect: connection refused
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:34 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a: {scylladbdatacenter-controller } StatefulSetUpdated: StatefulSet e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:34 +0000 UTC - event for basic-nztd8-us-east-1-us-east-1a-1: {scylladbdatacenter-controller } ServiceUpdated: Service e2e-test-scyllacluster-2ms98-0-85njt/basic-nztd8-us-east-1-us-east-1a-1 updated
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for basic-nztd8: {controllermanager } CalculateExpectedPodCountFailed: Failed to calculate the number of expected pods: jobs.batch does not implement the scale subresource
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0: {scylladbdatacenter-controller } JobCreated: Job e2e-test-scyllacluster-2ms98-0-85njt/cleanup-basic-nztd8-us-east-1-us-east-1a-0 created
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0: {job-controller } SuccessfulCreate: Created pod: cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp
  Aug 26 21:37:44.694: INFO: At 2024-08-26 21:37:44 +0000 UTC - event for cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp: {default-scheduler } Scheduled: Successfully assigned e2e-test-scyllacluster-2ms98-0-85njt/cleanup-basic-nztd8-us-east-1-us-east-1a-0-v4lbp to gke-so-3704e3ea-edbe-42a6-928-workers-f0cf17ac-47tl
  STEP: Collecting dumps from namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:44.694
  STEP: Destroying namespace "e2e-test-scyllacluster-2ms98-0-85njt". @ 08/26/24 21:37:45.437
  STEP: Waiting for namespace "e2e-test-scyllacluster-2ms98-0-85njt" to be removed. @ 08/26/24 21:37:45.449
  << Timeline
  [FAILED] Expected
      <[]string | len:5, cap:5>: ["10.17.250.51", "10.17.255.223", "10.125.193.50", "10.125.193.92", ""]
  to have length 4
  In [It] at: github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 @ 08/26/24 21:37:44.68
  Full Stack Trace
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func16.1.1({{0xc000cbbdc0?, 0x1?, 0x0?}, 0xc000bd6708?}, 0xc000a2dcf0, 0xc0007c7500, {0x35a2038, 0xc0005c5810}, {0xc0007f3040, 0x1, ...})
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:211 +0x21c9
    github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster.init.func16.1()
    	github.com/scylladb/scylla-operator/test/e2e/set/scyllacluster/scyllacluster_tls.go:259 +0x930

Looks like a cleanup Pod - triggered after considering ScyllaCluster as rolled out (it's an expected edge case) - got picked up by the test logic and IP of it (empty one) failed the assertion.
@zimnx zimnx added kind/flake Categorizes issue or PR as related to a flaky test. priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Aug 27, 2024
@zimnx zimnx mentioned this issue Aug 27, 2024
Merged
@scylladb scylladb deleted a comment from yldzmuhammed Aug 27, 2024
@scylla-operator-bot scylla-operator-bot
Copy link
Contributor

The Scylla Operator project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

  • After 30d of inactivity, lifecycle/stale is applied
  • After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
  • After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

  • Mark this issue as fresh with /remove-lifecycle stale
  • Close this issue with /close
  • Offer to help out

/lifecycle stale

@scylla-operator-bot scylla-operator-bot bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Sep 27, 2024
@zimnx
Copy link
Collaborator Author

zimnx commented Oct 2, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2137/pull-scylla-operator-master-e2e-gke-parallel/1841481213232025600

/remove-lifecycle stale

@scylla-operator-bot scylla-operator-bot bot removed the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 2, 2024
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2142/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1843940515221868544#1:test-build-log.txt%3A1226

@rzetelskik rzetelskik mentioned this issue Oct 10, 2024
Merged
1 task
@rzetelskik
Copy link
Member

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2156/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1846888608951177216#1:test-build-log.txt%3A1200

@rzetelskik rzetelskik mentioned this issue Oct 21, 2024
Merged
@zimnx
Copy link
Collaborator Author

zimnx commented Oct 22, 2024

https://prow.scylla-operator.scylladb.com/view/gs/scylla-operator-prow/pr-logs/pull/scylladb_scylla-operator/2160/pull-scylla-operator-master-e2e-gke-parallel-clusterip/1848755274836348928

@zimnx zimnx mentioned this issue Oct 22, 2024
Merged
@tnozicka tnozicka added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/important-longterm Important over the long term, but may not be staffed and/or may need multiple releases to complete. labels Oct 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@zimnx @tnozicka @rzetelskik