custom-metrics-stackdriver-adapter not able to read google managed prometheus metrics

[radioactive11]

I am trying to create an HPA based on prometheus metrics (google managed) but the HPA is not able to read those metrics.

This is the error log I am getting from the custom-metrics-stackdriver-adapter pods.

apiserver received an error that is not an metav1.Status: &googleapi.Error{Code:404, Message:"Cannot find metric(s) that match type = \"prometheus.googleapis.com/aries_backlog_pressure/gauge\" label = area label = pod. If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon.", Details:[]interface {}(nil), Body:"{\n  \"error\": {\n    \"code\": 404,\n    \"message\": \"Cannot find metric(s) that match type = \\\"prometheus.googleapis.com/aries_backlog_pressure/gauge\\\" label = area label = pod. If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon.\",\n    \"errors\": [\n      {\n        \"message\": \"Cannot find metric(s) that match type = \\\"prometheus.googleapis.com/aries_backlog_pressure/gauge\\\" label = area label = pod. If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon.\",\n        \"domain\": \"global\",\n        \"reason\": \"notFound\"\n      }\n    ],\n    \"status\": \"NOT_FOUND\"\n  }\n}\n", Header:http.Header{"Cache-Control":[]string{"private"}, "Content-Type":[]string{"application/json; charset=UTF-8"}, "Date":[]string{"Sat, 31 Aug 2024 05:06:44 GMT"}, "Server":[]string{"ESF"}, "Vary":[]string{"Origin", "X-Origin", "Referer"}, "X-Content-Type-Options":[]string{"nosniff"}, "X-Frame-Options":[]string{"SAMEORIGIN"}, "X-Xss-Protection":[]string{"0"}}, Errors:[]googleapi.ErrorItem{googleapi.ErrorItem{Reason:"notFound", Message:"Cannot find metric(s) that match type = \"prometheus.googleapis.com/aries_backlog_pressure/gauge\" label = area label = pod. If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon."}}, err:(*apierror.APIError)(0xc00222cfc0)}: googleapi: Error 404: Cannot find metric(s) that match type = "prometheus.googleapis.com/aries_backlog_pressure/gauge" label = area label = pod. If a metric was created recently, it could take up to 10 minutes to become available. Please try again soon., notFound

I am able to view these metrics on the GCP Monitoring page

This is my HPA configuration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: aries-celery-hpa
  namespace: aries
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: aries-celery
  minReplicas: 1
  maxReplicas: 2
  metrics:
  - type: External
    external:
      metric:
        name: prometheus.googleapis.com|aries_backlog_pressure|gauge
  
      target:
        type: AverageValue
        averageValue: 2

[CatherineF-dev]

If you are familiar with golang, could you check the value of filter here https://github.com/GoogleCloudPlatform/k8s-stackdriver/blob/master/custom-metrics-stackdriver-adapter/pkg/adapter/translator/query_builder.go#L804?

1. print this filter
2. rebuild this custom-metrics-stackdriver-adapter
3. check filter value
4. use the same filter to query this metric in Cloud Monitoring UI.