(willing to PR) Support deleting a metric label

[fzyzcjy]

Hi thanks for the project!

When working on the sgl-model-gateway in https://github.com/sgl-project/sglang, I would like to expose some metrics for each worker, e.g. the circuit breaker state / the health per worker. But the workers are added and deleted dynamically, and it is needed to remove the metric for the deleted worker, otherwise the values are weird (e.g. if we set health to zero, then we cannot know whether it is a unhealthy existing worker, or an already deleted worker. same for circuit breaker state - it use zero for closed state).

Looking at the code, I guess the simplest way may be to expose the Registry.delete_counter/delete_gauge/... methods, or alternatively expose the registry object. I am happy to PR if this looks good to you.

The Recency may not be great for this case, b/c firstly it uses a coarse mask but we may want to have more fine grained control. Secondly, if the gateway does not have incoming request for a while, we definitely do not want the metrics to be deleted, since all workers are still there.

[tobz]

👋🏻

This is a long-standing problem that I've tried a few times to find a satisfactory solution to and just haven't been able to crack the code, as it were. This is to say: I agree that a solution is needed, and I would be receptive to anyone looking to work on a solution. 😄

So we can expose methods to manually delete the metric handles from the registry, although depending on the exporter you're using, the registry might not be exposed, and it wouldn't make sense to expose it in the exporters in this repository... too much of an implementation detail. This is why, generally speaking, I've tried to attack this from the perspective of figuring out when a metric isn't "live" anymore.

Conceptually, this is making metrics function kind of like Arc<T>, where actual usages represent a strong reference, and the copy held by the registry is a weak reference. By treating them like this, it theoretically gives us a way to say "hey, this metric is no longer referenced by any code, so it's time to remove it and do final flushing/cleanup."

My problem with trying to implement this to date is that I haven't been able to find a satisfactory solution to the problem of needing to atomically check if a metric is "dead" (registry is the last reference) and remove it if so... things like Arc::try_unwrap give us the ability to consume the inner value if we're the last strong reference, but we also need to be able to remove the metric from the registry map at the same time, and this is where it gets tricky... since we need to consume the Arc which requires taking it out of the registry, even if we're not yet sure that no one else is holding the metric.

[fzyzcjy]

Hmm, as an initial version, what about we never really delete the metric, but simply checks whether we should expose it to the /metrics endpoint when the endpoint is being visited?

This is suboptimal because we do not release the underlying resource, but usually the cardinality over time should not be too high - otherwise Prometheus may be unhappy. For extra safety, we may warn users about this in docs.

Another naive way may be acquiring a lock when doing such modifications. The drawback is locks are slow, but I personally think it is really rare to add/delete a metric, i.e. doing a locking will not have noticable perf drop for the system.

[tobz]

I think the "soft delete" approach is doable, although you still end up needing to carry enough data around (like we do now with Generation<T>) that you almost might as well delete it in the first place.

I have some additional thoughts that I'll try to dump into the issue later today.

[fzyzcjy]

Sure, looking forward to it!

[fzyzcjy]

I have some additional thoughts that I'll try to dump into the issue later today.

A tiny gentle ping :)