[FEATURE] Process metrics endpoint

[imran2140]

Is your feature request related to a problem? Please describe.
Webhook broker processes keep received messages and jobs in memory until delivered. This may result in accumulation of large and varying number of messages and jobs in individual webhook-broker processes, depending on which messages received by the process and webhook broker configuration. To understand why a webhook broker process is using certain amount of memory, it would be helpful to have a metrics endpoint expose the relevant metrics for a webhook broker process.

Describe the solution you'd like
A HTTP /metrics endpoint that exposes number of messages, in-flight jobs for the webhook-broker process. Average message size (in bytes), total message size (in bytes) can also be exposed.

The output format can be in Prometheus / OpenMetrics text format, for potential integration with Prometheus compatible metrics scraping agents.

Prometheus Metrics Exposition formats
OpenMetrics Spec
(Datadog) Kubernetes Prometheus and OpenMetrics metrics collection
(Datadog) Mapping Prometheus Metrics to Datadog Metrics

Describe alternatives you've considered
The endpoint can support an optional format parameter to output the metrics in json format (format=json), which may be helpful for simple custom external checks, as json decoding tools / libraries are widely available.

Additional context
Given the endpoint is process specific, we may consider blocking access to the endpoint from the service endpoint / load balancer.