prometheus_remote_write: tenant_id dynamic templating

[lazam]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

The prometheus_remote_write sink does not support dynamic templating for the tenant_id field, unlike other sinks such as Loki. This prevents users from dynamically routing metrics to different tenants based on event data.

I've checked existing tests written in https://github.com/vectordotdev/vector/blob/bc654a79/src/sinks/prometheus/remote_write/tests.rs#L57 and https://github.com/vectordotdev/vector/blob/bc654a79/src/sinks/loki/integration_tests.rs#L437. Prometheus is configured statically while Loki isn't.

Attempted Solutions

I tried the following approaches but nothing works:

• Pass via variable
• Pass as a tag
• Pass via the tenant_id tag

transforms:  
  vector_metrics_remap:  
    type: remap  
    source: |  
      .o11y_tenant = "vector'
    
  kubernetes_metrics_remap:  
    type: remap  
    source: |  
      .tags.o11y_tenant = "k8s"

   kubernetes_forced_tenant_id:
    type: remap
    source: |
      .tags."__tenant_id__" = "o11y"

sinks: 
  prometheus_via_tags: 
    type: prometheus_remote_write 
    inputs: 
      - kubernetes_node_metrics_remap 
    endpoint: "http://grafana-mimir-gateway/api/v1/push" 
    tenant_id: "{{ tags.o11y_tenant }}"

  prometheus_via_vars: 
    type: prometheus_remote_write 
    inputs: 
      - vector_metrics_remap 
    endpoint: "http://grafana-mimir-gateway/api/v1/push" 
    tenant_id: "{{ o11y_tenant }}" 
    
  prometheus_via_forced_tags: 
    type: prometheus_remote_write 
    inputs: 
      - kubernetes_forced_tenant_id 
    endpoint: "http://grafana-mimir-gateway/api/v1/push" 

Proposal

Similar on how the template syntax works in general. Example on Loki sink:

  default_loki:
    type: loki
    inputs:
      - logs
    endpoint: "http://loki-gateway.logging.svc.cluster.local"
    tenant_id: "{{ o11y_tenant }}"

References

No response

Version

0.51.1