streaming chunk from store gateway failing #6497
Replies: 5 comments 18 replies
-
That error message makes it sound as if the store-gateway the querier was reading from has crashed. Could you please check for any issues in the store-gateway logs around the same time? |
Beta Was this translation helpful? Give feedback.
-
hi, there is no such error, but just below warnings in same sg logs: caller=memcached_client.go:335 level=warn name=chunks-cache msg="failed to fetch items from memcached" numKeys=45 firstKey=subrange:anonymous/01HDMT2JB20N7P7CHPA88D6KNN/chunks/000002:85744000:85760000 err="memcache: connect timeout to 10.7.27.130:11211" i'm running with below caching component: i'm trying to use stream to reduce the querier memory usage and start facing this issue. let me know, if you need more info. thanks. |
Beta Was this translation helpful? Give feedback.
-
The error message you posted contains a string like |
Beta Was this translation helpful? Give feedback.
-
i'm running on 2.10.0. |
Beta Was this translation helpful? Give feedback.
-
Hi @kadhamecha-conga. Hitting exactly the same error. Did you find the issue in your setup? Thanks |
Beta Was this translation helpful? Give feedback.
-
hi team,
I have enable below config to reduce querier memory usage , but on large range query is failing with below error:
execution: attempted to read series at index 16757 from stream, but the stream has failed: rpc error: code = Unavailable desc = closing transport due to: connection error: desc = "error reading from server: read tcp x.x.x.x:52180->x.x..x.x:9095: read: connection reset by peer", received prior goaway: code: NO_ERROR, debug data: "max_age"
below two values i have added as querier config:
prefer_streaming_chunks_from_store_gateways: true
streaming_chunks_per_store_gateway_series_buffer_size: 512
can you please help me to undersatnd where the issue is ?
what should be the value set for streaming_chunks_per_store_gateway_series_buffer_size , if query range is higher ?
thanks.
Beta Was this translation helpful? Give feedback.
All reactions