Support Custom Metric Attributes Per Request #6281
Conversation
krak3n
force-pushed
the
feat/dynamic-grpc-labels
branch
from
370d0b2
to
737f700
Compare
|
See also #6092 (which is still a draft). |
|
Could we follow the same pattern as we're doing for otelhttp? |
krak3n
force-pushed
the
feat/dynamic-grpc-labels
branch
from
737f700
to
e20e5cf
Compare
|
@dmathieu pushed up changes. Had problems getting the |
|
@dmathieu would the ability to access the We have a use case where we would only know what attributes to add to the metrics while we are in the handler after db / network calls. |
|
I don't know if we should make metrics so permissive. By their nature, metrics must be low cardinality. |
|
Yeah that's fair @dmathieu. How's this PR looking in general? |
| if f := c.MetricAttributesFn; f != nil { | ||
| attrs := f(ctx, rs.Payload) | ||
|
|
||
| gctx.metricAttrs = append(gctx.metricAttrs, attrs...) |
There was a problem hiding this comment.
It doesn't seem like we need to modify gctx here?
This also causes a race problem.
There was a problem hiding this comment.
So the #6092 also modified this, otherwise the metric attributes won't be propagated on the *stats.End call where the majority of the metrics are recorded, since I don't think *stats.End has access to the Payload although I will check that.
There was a problem hiding this comment.
InPayload is the only stats.RPCStats implementation that has access to the request payload. So the this is only time we can call the MetricAttributesFn to be able to append too gRPCContext.metricAttrs, I can put a lock around metricAttrs protect against races?
There was a problem hiding this comment.
I'm very hesitant about updating a pointer to struct that is stored as a context value. That really looks like a smell.
Unfortunately, we can't do things cleanly (provide a new context with the updated value), as we can't provide a new context as return value.
There was a problem hiding this comment.
Ok, this doesn't seem like it's going to be possible then due to the limitations in how the stats.Handler works.
There was a problem hiding this comment.
@dmathieu any thoughts on if we should continue with this or not?
There was a problem hiding this comment.
I'd like to have other opinions.
There was a problem hiding this comment.
Ok, I'll leave it with you @dmathieu 👍
I did, but I only had issues with one test as I mentioned here: #6281 (comment) |
krak3n
changed the title
What
Adds a new
WithMetricsAttributesFnoption which allows custom metric attributes to be added on a per request basis.Why
So standard metrics provided by the gRPC instrumentation package can be annotated on a per request basis. Our use case is to add attributes to the instruments depending on what the handler is doing with the request.