Add native histogram for tracking push requests size #6384
Conversation
SungJin1212
force-pushed
the
Add-histogram-for-tracking-various-push-size
branch
4 times, most recently
from
09cc2c9 to
98b4613
Compare
SungJin1212
force-pushed
the
Add-histogram-for-tracking-various-push-size
branch
from
98b4613 to
5b4c6d8
Compare
| } | ||
|
|
||
| func (m *PushHandlerMetrics) ObservePushRequestSize(user, format string, size float64) { | ||
| if m != nil { |
| NativeHistogramBucketFactor: 1.1, | ||
| NativeHistogramMinResetDuration: 1 * time.Hour, | ||
| NativeHistogramMaxBucketNumber: 100, | ||
| }, []string{"user", "format"}), |
There was a problem hiding this comment.
I wonder if we really need the format parameter here. Why would we need different limits for different formats? Do we plan to have a new limit for RW 2.0?
I feel we should consolidate the request size limit into one #6333.
There was a problem hiding this comment.
I have no plan to something limit for PRW 2.0. It could be that we can add some limits. But, I thought different formats (prw1, prw2, and otlp) would have different body sizes.
So, I attached the format label so that the user can configure limit configs for different formats.
There was a problem hiding this comment.
If we want to track different formats then for the same reason we need to differentiate compression formats as well as they have different sizes. OTLP has protobuf and JSON then we also need to differentiate that.
To simplify things I think it is good enough to use a single size limit and metric
There was a problem hiding this comment.
It makes sense.
Then, de-attach format and make 'cortex_distributor_push_requests_uncompressed_size_bytestrack the uncompressed size of all of the formats. Next step is to deletedistributor.otlp-max-recv-msg-size` flags.
There was a problem hiding this comment.
Let's hear from other maintainers...
@friedrichg @danielblando @alanprot @CharlieTLe ?
SungJin1212
force-pushed
the
Add-histogram-for-tracking-various-push-size
branch
2 times, most recently
from
24f6351 to
69eb3fb
Compare
Signed-off-by: SungJin1212 <[email protected]>
SungJin1212
force-pushed
the
Add-histogram-for-tracking-various-push-size
branch
from
69eb3fb to
b813e04
Compare
This PR adds a
cortex_distributor_push_requests_uncompressed_bytesnative histogram to track uncompressed push requests in bytes for tenant and format.Two push formats exist now and 1 more push format will be added (Remote Write v2).
We are converting these things and addressing them as a
Remote Write1 Formatin internal logic.So, we cannot track each push request's behavior on the Distributor.
To address this concern, I added
PushHandlerMetricsto gather metrics at the push handler.I introduce
cortex_distributor_push_requests_uncompressed_bytesfirst; it can give the user insight into configuring the values of-distributor.max-recv-msg-sizeand-distributor.otlp-max-recv-msg-size.Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]