Skip to content

Conversation

@openshift-bot
Copy link
Contributor

@openshift-bot openshift-bot commented Dec 5, 2025

disruptions Information

There were (3616) added jobs and (344) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 285 jobs increased and 800 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 732 4m55.8s 495.17% 7m24.97s 791.01%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 732 5m3.25s 327.11% 7m33.66s 430.22%
ingress-to-oauth-server-new-connections 4.21 s390x ovn none ha 156 4m58.25s 175.90% 13m22.3s 62.67%
ingress-to-console-new-connections 4.21 s390x ovn none ha 156 4m58.25s 175.90% 13m22.85s 62.78%
cache-kube-api-new-connections 4.21 s390x ovn none ha 156 4m48s 183.19% 13m22.95s 62.84%
kube-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 183.28% 13m23.5s 62.96%
cache-openshift-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 182.45% 13m23.5s 62.96%
openshift-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 182.45% 13m22.95s 62.84%
oauth-api-new-connections 4.21 s390x ovn none ha 156 4m46.5s 181.71% 13m23.5s 62.96%
cache-oauth-api-new-connections 4.21 s390x ovn none ha 156 4m46.5s 181.71% 13m23.5s 62.96%
metrics-api-new-connections 4.21 s390x ovn none ha 156 4m47s 140.57% 13m22.3s 62.77%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 732 3m19s 215.87% 4m37s 265.53%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 732 3m11.45s 203.89% 4m27.76s 253.34%
metrics-api-reused-connections 4.21 s390x ovn none ha 156 3m1s 229.69% 13m14.8s 62.28%
ingress-to-console-reused-connections 4.21 s390x ovn none ha 156 3m15.75s 135.28% 13m16.35s 62.06%
ingress-to-oauth-server-reused-connections 4.21 s390x ovn none ha 156 3m15.5s 134.13% 13m15.8s 61.95%
cache-openshift-api-reused-connections 4.21 s390x ovn none ha 156 2m4.5s 128.86% 13m15.25s 62.03%
cache-oauth-api-reused-connections 4.21 s390x ovn none ha 156 2m14.25s 107.82% 13m15.25s 62.01%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 112 1m27.75s 379.51% 7m27.72s 589.65%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 112 1m9s 2364.29% 4m55.72s 437.48%
cache-kube-api-reused-connections 4.21 s390x ovn none ha 156 2m9s 105.09% 13m15.25s 62.03%
openshift-api-reused-connections 4.21 s390x ovn none ha 156 2m8.75s 103.72% 13m15.8s 62.14%
oauth-api-reused-connections 4.21 s390x ovn none ha 156 2m7.75s 102.14% 13m15.8s 62.14%
kube-api-reused-connections 4.21 s390x ovn none ha 156 2m6.75s 100.55% 13m15.25s 62.03%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 54.15s 2607.50% 2m49.38s 82.72%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn azure ha 684 1m25s 86.00% 1m33.17s 102.81%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 41.05s 1125.37% 1m37.67s 34.51%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 66.78% 2m8.76s 186.32%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 63.14% 2m9.96s 182.71%
kube-api-http1-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 63.14% 2m9.96s 182.71%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m12.8s 62.32% 2m6.84s 182.05%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 11s 307.41% 48.92s 1563.95%
kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 8s 700.00% 23s 248.48%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 684 5s 400.00% 7s 600.00%
service-load-balancer-with-pdb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 3s 50.00% 4s 100.00%
kube-api-http1-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2s 106.19%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1112 1s 0.00% 2s 100.00%
kube-api-http1-internal-lb-reused-connections 4.21 amd64 ovn metal compact 107 8m21.6s 0.00% 17m25.92s 43.20%
kube-api-http1-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 732 1s 0.00% 3.69s 314.61%
metrics-api-reused-connections 4.21 4.21 amd64 ovn aws ha 732 1s 0.00% 16.69s 1775.28%
cache-openshift-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
cache-kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 25s 0.00% 40.38s 1034.27%
openshift-api-http2-service-network-reused-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 49.8s 0.00% 1m38s 233.33%
openshift-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
cache-oauth-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
image-registry-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2s 106.19%
image-registry-reused-connections 4.21 4.20 amd64 ovn gcp ha 763 1s 0.00% 2s 100.00%
metrics-api-new-connections 4.21 4.21 amd64 ovn aws ha 732 14s 0.00% 28.38s 3088.76%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 2m32.78s 233.51%
host-to-service-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 2s 100.00%
kube-api-http1-service-network-reused-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1193 2s 0.00% 3m54.64s 686.06%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1112 4s 0.00% 1m37.56s 31.45%
kube-api-http1-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 12s 0.00% 35s 3400.00%
pod-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 732 0s 0.00% 4s 43.88%
openshift-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1193 2s 0.00% 8.08s 34.67%
metrics-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2.96s 196.00%
cache-oauth-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
openshift-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 107 20m11.1s 0.00% 34m4.3s 35.65%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1112 1s 0.00% 4m13.78s 231.39%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn aws ha 732 47.45s 0.00% 1m56.9s 3944.98%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 25s 0.00% 37s 1180.28%
kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 94.00%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 1s 170.27%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 156 3s 0.00% 2m55.15s 6110.99%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1193 3s 0.00% 5m18.92s 519.62%
pod-to-service-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 50ms 0.00% 1.01s 380.95%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 4m11.96s 728.82%
kube-api-http1-service-network-reused-connections 4.21 amd64 ovn metal compact 107 10m4.7s 0.00% 18m46.3s 50.99%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 2m2.35s 0.00% 3m36.76s 4332.72%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
oauth-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1193 0s 0.00% 7s 40.00%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 16s 0.00% 25s 2400.00%
cache-oauth-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 21.96s 2163.92%
openshift-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 45s 0.00% 1m40.88s 4944.00%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 16s 0.00% 25.96s 2496.00%
pod-to-service-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 18.88s 1846.39%
openshift-api-new-connections 4.21 4.20 amd64 ovn aws ha 1193 0s 0.00% 8s 38.65%
host-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 732 0s 0.00% 5s 415.46%
kube-api-http1-service-network-new-connections 4.21 amd64 ovn metal compact 107 17m48.5s 0.00% 30m54.86s 31.02%
openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 20.8s 0.00% 53.72s 53.31%

alerts Information

There were (15045) added jobs and (2006) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 40 jobs increased and 179 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
ClusterOperatorDegraded openshift-cluster-version Warning 4.21 4.21 amd64 ovn gcp ha 272 1h40m31.9s 221.34% 2h10m39.9s 254.57%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn vsphere ha 281 1h26m24s 126.53% 1h51m24.4s 75.48%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn aws ha 311 17m24s 105.11% 20m40.4s 112.18%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn aws ha 483 14m19.2s 47.93% 22m44.16s 59.02%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 174 6m37.8s 166.09% 13m43.66s 166.21%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn gcp ha 209 12m1.2s 46.73% 19m57.68s 71.46%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 281 27m26s 0.00% 34m28s 46060.71%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 311 1m58s 0.00% 3m51.4s 384.10%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 483 0s 0.00% 4m33.04s 370.76%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 174 13m32.1s 0.00% 22m8.72s 33.61%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn aws single 129 29m24.4s 0.00% 50m10.4s 33.83%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn aws single 129 1h4m36.8s 0.00% 2h3m41.28s 36.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 272 28s 0.00% 2m28s 428.57%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.21 amd64 ovn aws ha 311 2m58s 0.00% 6m53s 132.02%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn aws single 129 7m46s 0.00% 18m40.4s 90.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 174 0s 0.00% 1m26.54s 5747.30%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 483 0s 0.00% 3m3.04s 249.58%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 393 35.2s 0.00% 9m56s 188.62%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 209 15m52s 0.00% 30m34.4s 88.92%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn vsphere ha 281 38m28s 0.00% 1h36m46.8s 375.90%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 256 2m4s 0.00% 12m1.6s 90.09%
/cc @xueqzhan, @neisw, @smg247

@openshift-ci-robot
Copy link

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 5, 2025
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

In response to this:

disruptions Information

There were (2748) added jobs and (0) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 136 jobs increased and 891 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 483 4m31.5s 684.68% 7m46.36s 30.05%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn aws single 483 3m3.7s 652.87% 5m3s 30.24%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 104 46.7s 1567.86% 4m59.56s 444.46%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn azure ha 551 1m17s 104.24% 1m26.5s 127.99%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 104 48.5s 165.03% 7m31.56s 595.56%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 577 1m18s 56.94% 3m31.04s 322.59%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 577 1m6s 49.66% 2m10s 190.05%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1m0s 33.78% 1m11.96s 60.02%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 1m0s 30.86% 1m12s 56.62%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 551 5s 400.00% 9s 800.00%
kube-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 551 5s 85.19% 7s 138.10%
host-to-host-new-connections 4.21 4.20 amd64 ovn gcp ha 618 0s 0.00% 1s 78.57%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn aws ha 577 0s 0.00% 54.12s 1772.66%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 4m11.99s 728.91%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 0s 0.00% 12.89s 1189.00%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 0s 0.00% 14s 1300.00%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn azure ha 551 2s 0.00% 14s 104.08%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 4m13.42s 230.92%
cache-openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 2s 100.00%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1130 2s 0.00% 2m11.95s 77.78%
host-to-service-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 1.71s 71.00%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1188 1s 0.00% 5m21.37s 524.38%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 136 250ms 0.00% 1m5.15s 2210.28%
openshift-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 45s 415.46%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 577 1m6.2s 0.00% 2m9.24s 70.55%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1202 1m8s 0.00% 1m34.9s 49.19%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 2m32.42s 232.72%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 10.97s 273.13%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1130 1s 0.00% 2s 100.00%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 577 1m16.2s 0.00% 3m38.68s 155.59%
image-registry-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 2s 106.19%
openshift-api-http2-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 577 7.2s 0.00% 2m1.44s 189.63%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn azure ha 1130 0s 0.00% 1s 170.27%
kube-api-http2-service-network-reused-connections 4.21 amd64 ovn aws ha 202 0s 0.00% 990ms 253.57%
oauth-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1202 1s 0.00% 3s 209.28%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1188 1s 0.00% 3m55.04s 687.40%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 180 0s 0.00% 1s 150.00%
openshift-api-reused-connections 4.21 4.21 amd64 ovn aws ha 577 2s 0.00% 56.44s 1781.33%

alerts Information

There were (10014) added jobs and (1300) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 31 jobs increased and 195 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
ClusterOperatorDegraded openshift-cluster-version Warning 4.21 4.21 amd64 ovn gcp ha 279 2h19m53.4s 347.15% 2h25m27.08s 294.69%
TargetDown openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 139 1h57m36.6s 55.54% 2h13m1.8s 49.95%
KubeAggregatedAPIDown default Warning 4.21 4.21 amd64 ovn aws single 166 22m12s 155.17% 35m55.5s 36.16%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 139 3m31s 41.14% 11m57.24s 131.82%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 139 0s 0.00% 2m28s 9900.00%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn vsphere ha 225 3h11m7.2s 0.00% 6h22m36.96s 64.15%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 196 28s 0.00% 2m1s 153.14%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 279 2.8s 0.00% 2m28s 428.57%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 354 0s 0.00% 3m39.24s 278.00%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 225 0s 0.00% 4m21.52s 5737.50%
OVNKubernetesResourceRetryFailure openshift-ovn-kubernetes Warning 4.21 amd64 ovn metal ha 308 0s 0.00% 9m28s 176.53%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 158 2m16s 0.00% 27m7.3s 67.59%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 308 0s 0.00% 8m13.4s 138.93%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn azure ha 139 0s 0.00% 10m30.24s 474.30%
/cc @xueqzhan, @neisw, @smg247

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested review from deads2k and sjenning December 5, 2025 01:07
@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@neisw
Copy link
Contributor

neisw commented Dec 5, 2025

/lgtm
/verified bypass
/retest-required

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Dec 5, 2025
@openshift-ci-robot
Copy link

@neisw: The verified label has been added.

In response to this:

/lgtm
/verified bypass
/retest-required

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Dec 5, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 5, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: neisw, openshift-bot

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Dec 5, 2025
@openshift-bot openshift-bot force-pushed the no-issue--automated---update-synthetic-test-data branch from 27d6341 to 19745d6 Compare December 12, 2025 01:03
@openshift-ci-robot openshift-ci-robot removed the verified Signifies that the PR passed pre-merge verification criteria label Dec 12, 2025
@openshift-ci-robot
Copy link

@openshift-bot: This pull request explicitly references no jira issue.

In response to this:

disruptions Information

There were (3616) added jobs and (344) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, disruptions had 285 jobs increased and 800 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 732 4m55.8s 495.17% 7m24.97s 791.01%
openshift-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 732 5m3.25s 327.11% 7m33.66s 430.22%
ingress-to-oauth-server-new-connections 4.21 s390x ovn none ha 156 4m58.25s 175.90% 13m22.3s 62.67%
ingress-to-console-new-connections 4.21 s390x ovn none ha 156 4m58.25s 175.90% 13m22.85s 62.78%
cache-kube-api-new-connections 4.21 s390x ovn none ha 156 4m48s 183.19% 13m22.95s 62.84%
kube-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 183.28% 13m23.5s 62.96%
cache-openshift-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 182.45% 13m23.5s 62.96%
openshift-api-new-connections 4.21 s390x ovn none ha 156 4m47.25s 182.45% 13m22.95s 62.84%
oauth-api-new-connections 4.21 s390x ovn none ha 156 4m46.5s 181.71% 13m23.5s 62.96%
cache-oauth-api-new-connections 4.21 s390x ovn none ha 156 4m46.5s 181.71% 13m23.5s 62.96%
metrics-api-new-connections 4.21 s390x ovn none ha 156 4m47s 140.57% 13m22.3s 62.77%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn aws ha 732 3m19s 215.87% 4m37s 265.53%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn aws ha 732 3m11.45s 203.89% 4m27.76s 253.34%
metrics-api-reused-connections 4.21 s390x ovn none ha 156 3m1s 229.69% 13m14.8s 62.28%
ingress-to-console-reused-connections 4.21 s390x ovn none ha 156 3m15.75s 135.28% 13m16.35s 62.06%
ingress-to-oauth-server-reused-connections 4.21 s390x ovn none ha 156 3m15.5s 134.13% 13m15.8s 61.95%
cache-openshift-api-reused-connections 4.21 s390x ovn none ha 156 2m4.5s 128.86% 13m15.25s 62.03%
cache-oauth-api-reused-connections 4.21 s390x ovn none ha 156 2m14.25s 107.82% 13m15.25s 62.01%
azure-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 112 1m27.75s 379.51% 7m27.72s 589.65%
gcp-network-liveness-new-connections 4.21 amd64 ovn vsphere ha 112 1m9s 2364.29% 4m55.72s 437.48%
cache-kube-api-reused-connections 4.21 s390x ovn none ha 156 2m9s 105.09% 13m15.25s 62.03%
openshift-api-reused-connections 4.21 s390x ovn none ha 156 2m8.75s 103.72% 13m15.8s 62.14%
oauth-api-reused-connections 4.21 s390x ovn none ha 156 2m7.75s 102.14% 13m15.8s 62.14%
kube-api-reused-connections 4.21 s390x ovn none ha 156 2m6.75s 100.55% 13m15.25s 62.03%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 54.15s 2607.50% 2m49.38s 82.72%
openshift-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn azure ha 684 1m25s 86.00% 1m33.17s 102.81%
gcp-network-liveness-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 41.05s 1125.37% 1m37.67s 34.51%
kube-api-http2-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 66.78% 2m8.76s 186.32%
kube-api-http2-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 63.14% 2m9.96s 182.71%
kube-api-http1-localhost-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m14.8s 63.14% 2m9.96s 182.71%
kube-api-http1-localhost-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1m12.8s 62.32% 2m6.84s 182.05%
openshift-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 11s 307.41% 48.92s 1563.95%
kube-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 8s 700.00% 23s 248.48%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn azure ha 684 5s 400.00% 7s 600.00%
service-load-balancer-with-pdb-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 3s 50.00% 4s 100.00%
kube-api-http1-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2s 106.19%
openshift-api-http2-internal-lb-reused-connections 4.21 4.20 amd64 ovn azure ha 1112 1s 0.00% 2s 100.00%
kube-api-http1-internal-lb-reused-connections 4.21 amd64 ovn metal compact 107 8m21.6s 0.00% 17m25.92s 43.20%
kube-api-http1-internal-lb-reused-connections 4.21 4.21 amd64 ovn aws ha 732 1s 0.00% 3.69s 314.61%
metrics-api-reused-connections 4.21 4.21 amd64 ovn aws ha 732 1s 0.00% 16.69s 1775.28%
cache-openshift-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
cache-kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 25s 0.00% 40.38s 1034.27%
openshift-api-http2-service-network-reused-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
openshift-api-http2-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 49.8s 0.00% 1m38s 233.33%
openshift-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
cache-oauth-api-new-connections 4.21 amd64 ovn azure external 786 0s 0.00% 1.45s 45.00%
image-registry-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2s 106.19%
image-registry-reused-connections 4.21 4.20 amd64 ovn gcp ha 763 1s 0.00% 2s 100.00%
metrics-api-new-connections 4.21 4.21 amd64 ovn aws ha 732 14s 0.00% 28.38s 3088.76%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 2m32.78s 233.51%
host-to-service-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 2s 100.00%
kube-api-http1-service-network-reused-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
gcp-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1193 2s 0.00% 3m54.64s 686.06%
pod-to-host-new-connections 4.21 4.20 amd64 ovn azure ha 1112 4s 0.00% 1m37.56s 31.45%
kube-api-http1-service-network-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 12s 0.00% 35s 3400.00%
pod-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 732 0s 0.00% 4s 43.88%
openshift-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1193 2s 0.00% 8.08s 34.67%
metrics-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 2.96s 196.00%
cache-oauth-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 74.77%
openshift-api-http2-internal-lb-new-connections 4.21 amd64 ovn metal compact 107 20m11.1s 0.00% 34m4.3s 35.65%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn azure ha 1112 1s 0.00% 4m13.78s 231.39%
kube-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn aws ha 732 47.45s 0.00% 1m56.9s 3944.98%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 25s 0.00% 37s 1180.28%
kube-api-reused-connections 4.21 amd64 ovn metal compact 107 1s 0.00% 1.94s 94.00%
openshift-api-http2-internal-lb-new-connections 4.21 4.20 amd64 ovn azure ha 1112 0s 0.00% 1s 170.27%
azure-network-liveness-new-connections 4.21 s390x ovn none ha 156 3s 0.00% 2m55.15s 6110.99%
azure-network-liveness-new-connections 4.21 4.20 amd64 ovn aws ha 1193 3s 0.00% 5m18.92s 519.62%
pod-to-service-new-connections 4.21 4.21 amd64 ovn vsphere ha 100 50ms 0.00% 1.01s 380.95%
azure-network-liveness-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 4m11.96s 728.82%
kube-api-http1-service-network-reused-connections 4.21 amd64 ovn metal compact 107 10m4.7s 0.00% 18m46.3s 50.99%
openshift-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn aws ha 732 2m2.35s 0.00% 3m36.76s 4332.72%
aws-network-liveness-new-connections 4.21 multi ovn aws ha 218 0s 0.00% 830ms 107.50%
oauth-api-reused-connections 4.21 4.20 amd64 ovn aws ha 1193 0s 0.00% 7s 40.00%
kube-api-http1-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 16s 0.00% 25s 2400.00%
cache-oauth-api-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 21.96s 2163.92%
openshift-api-http2-service-network-reused-connections 4.21 4.21 amd64 ovn gcp ha 1205 45s 0.00% 1m40.88s 4944.00%
kube-api-http2-internal-lb-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 16s 0.00% 25.96s 2496.00%
pod-to-service-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 1s 0.00% 18.88s 1846.39%
openshift-api-new-connections 4.21 4.20 amd64 ovn aws ha 1193 0s 0.00% 8s 38.65%
host-to-host-new-connections 4.21 4.21 amd64 ovn aws ha 732 0s 0.00% 5s 415.46%
kube-api-http1-service-network-new-connections 4.21 amd64 ovn metal compact 107 17m48.5s 0.00% 30m54.86s 31.02%
openshift-api-new-connections 4.21 4.21 amd64 ovn gcp ha 1205 20.8s 0.00% 53.72s 53.31%

alerts Information

There were (15045) added jobs and (2006) were removed.

Comparisons were above allowed leeway of 30.00%

Note: For P99, alerts had 40 jobs increased and 179 jobs decreased.

Click To Show Table
Name Release From Arch Network Platform Topology Job Results P95 P95 % Increase P99 P99 % Increase
ClusterOperatorDegraded openshift-cluster-version Warning 4.21 4.21 amd64 ovn gcp ha 272 1h40m31.9s 221.34% 2h10m39.9s 254.57%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn vsphere ha 281 1h26m24s 126.53% 1h51m24.4s 75.48%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 4.21 amd64 ovn aws ha 311 17m24s 105.11% 20m40.4s 112.18%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn aws ha 483 14m19.2s 47.93% 22m44.16s 59.02%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 174 6m37.8s 166.09% 13m43.66s 166.21%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn gcp ha 209 12m1.2s 46.73% 19m57.68s 71.46%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn vsphere ha 281 27m26s 0.00% 34m28s 46060.71%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn aws ha 311 1m58s 0.00% 3m51.4s 384.10%
HighOverallControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 483 0s 0.00% 4m33.04s 370.76%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 arm64 ovn aws ha 174 13m32.1s 0.00% 22m8.72s 33.61%
HighOverallControlPlaneMemory openshift-machine-config-operator Warning 4.21 amd64 ovn aws single 129 29m24.4s 0.00% 50m10.4s 33.83%
KubeJobFailed openshift-insights Warning 4.21 amd64 ovn aws single 129 1h4m36.8s 0.00% 2h3m41.28s 36.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 4.21 amd64 ovn gcp ha 272 28s 0.00% 2m28s 428.57%
CsvAbnormalFailedOver2Min openshift-operator-lifecycle-manager Warning 4.21 4.21 amd64 ovn aws ha 311 2m58s 0.00% 6m53s 132.02%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn aws single 129 7m46s 0.00% 18m40.4s 90.87%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 arm64 ovn aws ha 174 0s 0.00% 1m26.54s 5747.30%
ExtremelyHighIndividualControlPlaneCPU openshift-kube-apiserver Warning 4.21 amd64 ovn aws ha 483 0s 0.00% 3m3.04s 249.58%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn metal ha 393 35.2s 0.00% 9m56s 188.62%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn gcp ha 209 15m52s 0.00% 30m34.4s 88.92%
NodeSystemSaturation openshift-monitoring Warning 4.21 amd64 ovn vsphere ha 281 38m28s 0.00% 1h36m46.8s 375.90%
NodeSystemSaturation openshift-monitoring Warning 4.21 4.21 amd64 ovn azure ha 256 2m4s 0.00% 12m1.6s 90.09%
/cc @xueqzhan, @neisw, @smg247

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot removed the lgtm Indicates that a PR is ready to be merged. label Dec 12, 2025
@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 12, 2025

New changes are detected. LGTM label has been removed.

@openshift-ci-robot
Copy link

Scheduling required tests:
/test e2e-aws-csi
/test e2e-aws-ovn-fips
/test e2e-aws-ovn-microshift
/test e2e-aws-ovn-microshift-serial
/test e2e-aws-ovn-serial-1of2
/test e2e-aws-ovn-serial-2of2
/test e2e-gcp-csi
/test e2e-gcp-ovn
/test e2e-gcp-ovn-upgrade
/test e2e-metal-ipi-ovn-ipv6
/test e2e-vsphere-ovn
/test e2e-vsphere-ovn-upi

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Dec 12, 2025

@openshift-bot: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-gcp-ovn 19745d6 link true /test e2e-gcp-ovn
ci/prow/e2e-aws-ovn-serial-1of2 19745d6 link true /test e2e-aws-ovn-serial-1of2

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants