[BUG] Extremely slow after upgrading to 2.17.1 dashboards and server

[adrianmelero-madiva]

After upgrading my cluster to 2.17.1 (dashboards and server), searches from dashboards are extremely slow, but other integrations like grafana datasource opensearch are searching fast.

[Hailong-am]

After upgrading my cluster to 2.17.1 (dashboards and server), searches from dashboards are extremely slow, but other integrations like grafana datasource opensearch are searching fast.

@adrianmelero-madiva can you provide more details? like which pages/apis you surface with slow response? Also you can try to query opensearch directly to make sure it not comes from OpenSearch itself?

[adrianmelero-madiva]

if I query opensearch directly it works fine and fast, in dashboards opening the default index in discover takes lonkg time. Here a sample of slow requests. It seems to be related with long numerals. You can see response time of 6 seconds, other request goes to 9 and 10 seconds.

{"type":"response","@timestamp":"2024-11-12T10:04:02Z","tags":[],"pid":1,"method":"post","statusCode":200,"req":{"url":"/internal/search/opensearch-with-long-numerals","method":"post","headers":{"host":"opensearch-dashboards...*","content-length":"958","osd-version":"2.17.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","content-type":"application/json","osd-xsrf":"osd-fetch","accept":"/","origin":"http://opensearch-dashboards.****.****.****","referer":"http://opensearch-dashboards.****.****.***/app/data-explorer/discover","accept-encoding":"gzip, deflate","accept-language":"es-ES,es;q=0.9","x-forwarded-for":"10.50.48.153","x-forwarded-proto":"http","x-request-id":"7f4e727f-841a-4973-b6e1-738a6aed1d10","x-envoy-attempt-count":"1","x-envoy-internal":"true","x-forwarded-client-cert":"By=spiffe://cluster.local/ns/default/sa/opensearchstack-dashboards;Hash=ff7617a084f378c8530410d0952e31204b3aced0e07bd4fac7bf64f0fb8f81ed;Subject="";URI=spiffe://cluster.local/ns/istio-ingress/sa/default","x-b3-traceid":"e123c1b8a4b6be21f197d17171bc3f49","x-b3-spanid":"53bd83bc2aab808d","x-b3-parentspanid":"f197d17171bc3f49","x-b3-sampled":"0","securitytenant":""},"remoteAddress":"127.0.0.6","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36","referer":"http://opensearch-dashboards.****.****.***/app/data-explorer/discover"},"res":{"statusCode":200,"responseTime":6006,"contentLength":9},"message":"POST /internal/search/opensearch-with-long-numerals 200 6006ms - 9.0B"}

[Hailong-am]

@adrianmelero-madiva before you upgrade to 2.17.1 what's the older version? and what's the result number of documents returned by this query. it might related to long numerals.

you can do a quick update on your opensearch dashboard distribution with file

opensearch-dashboards-2.17.1/node_modules/@opensearch-project/opensearch-next/lib/Serializer.js

find val < Number.MAX_SAFE_INTEGER at line 104 and replace it with val < Number.MIN_SAFE_INTEGER and then restart your OSD to see whether the performance is improved or not.