[Keda-Datadog Scaler] Error establishing Datadog connection - 403 Forbidden connecting to Datadog API endpoint #6408

senthilprasanth · 2024-12-08T00:25:03Z

senthilprasanth
Dec 8, 2024

Report

Encountering "Error establishing Datadog connection - 403 Forbidden connecting to Datadog API endpoint".
We have summarized the issue with details as much as possible here to your convenience.

KBs used for implementation

https://keda.sh/docs/2.16/scalers/datadog/
https://keda.sh/docs/2.16/concepts/authentication/
https://keda.sh/docs/2.16/authentication-providers/aws-secret-manager/

Expected Behavior

Keda Datadog scaler to be activated

Actual Behavior

"error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"

Steps to reproduce the problem

Install Keda to your EKS cluster
Create secrets at AWS secret manager and integrate it with ClusterTriggerAuthentication
Deploy CRDs "ClusterTriggerAuthentication" and "ScaledObject"

Logs from KEDA operator

Enabled Debug mode at keda operator

ERROR	scale_handler	error resolving auth params	{"type": "ScaledObject", "namespace": "keda", "name": "accounts-investor-accounts-api", "triggerIndex": 0, "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}
ERROR	Failed to create new HPA resource	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"accounts-investor-accounts-api","namespace":"keda"}, "namespace": "keda", "name": "accounts-investor-accounts-api", "reconcileID": "e3257394-90b4-4a26-a03f-5980c6938316", "HPA.Namespace": "keda", "HPA.Name": "keda-hpa-accounts-investor-accounts-api", "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}
ERROR	failed to ensure HPA is correctly created for ScaledObject	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"accounts-investor-accounts-api","namespace":"keda"}, "namespace": "keda", "name": "accounts-investor-accounts-api", "reconcileID": "e3257394-90b4-4a26-a03f-5980c6938316", "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}
DEBUG	events	error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden	{"type": "Warning", "object": {"kind":"ScaledObject","namespace":"keda","name":"accounts-investor-accounts-api","uid":"784b1211-e7bd-4a10-b0a2-d2bbbc93451c","apiVersion":"keda.sh/v1alpha1","resourceVersion":"355403543"}, "reason": "KEDAScalerFailed"}
DEBUG	events	failed to ensure HPA is correctly created for ScaledObject	{"type": "Warning", "object": {"kind":"ScaledObject","namespace":"keda","name":"accounts-investor-accounts-api","uid":"784b1211-e7bd-4a10-b0a2-d2bbbc93451c","apiVersion":"keda.sh/v1alpha1","resourceVersion":"355403543"}, "reason": "ScaledObjectCheckFailed"}
ERROR	Reconciler error	{"controller": "scaledobject", "controllerGroup": "keda.sh", "controllerKind": "ScaledObject", "ScaledObject": {"name":"accounts-investor-accounts-api","namespace":"keda"}, "namespace": "keda", "name": "accounts-investor-accounts-api", "reconcileID": "e3257394-90b4-4a26-a03f-5980c6938316", "error": "error establishing Datadog connection: error connecting to Datadog API endpoint: 403 Forbidden"}}

KEDA version

2.16.0

Kubernetes Version

1.30.0

Platform

AWS EKS

Scaler Details

Datadog

cURL check

We could successfully cURL the DD API endpoint from my local machine

curl -v -X GET "https://api.datadoghq.com/api/v1/validate" -H "Accept: application/json" -H "DD-API-KEY: XXXXX"
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 3.233.154.1:443...
* Connected to api.datadoghq.com (3.233.154.1) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: C=US; ST=New York; L=New York; CN=*.api.datadoghq.com; O=Zscaler Inc.; OU=Zscaler Inc.
*  start date: Dec  7 19:45:06 2024 GMT
*  expire date: Dec 21 02:39:56 2024 GMT
*  subjectAltName: host "api.datadoghq.com" matched cert's "api.datadoghq.com"
*  issuer: C=US; ST=California; O=Zscaler Inc.; OU=Zscaler Inc.; CN=Zscaler Intermediate Root CA (zscaler.net) (t)
*  SSL certificate verify ok.
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET /api/v1/validate HTTP/1.1
> Host: api.datadoghq.com
> User-Agent: curl/7.85.0
> Accept: application/json
> DD-API-KEY: XXXX
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Date: Sat, 07 Dec 2024 23:36:39 GMT
< Content-Type: application/json
< Content-Length: 14
< Connection: keep-alive
< content-security-policy: frame-ancestors 'self'; report-uri https://logs.browser-intake-datadoghq.com/api/v2/logs?dd-api-key=pube4f163c23bbf91c16b8f57f56af9fc58&dd-evp-origin=content-security-policy&ddsource=csp-report&ddtags=site%3Adatadoghq.com
< vary: Accept-Encoding
< x-frame-options: SAMEORIGIN
< x-content-type-options: nosniff
< strict-transport-security: max-age=31536000; includeSubDomains; preload
<
* Connection #0 to host api.datadoghq.com left intact

Config used

ClusterTriggerAuthentication

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-clustertriggerauth
spec:
  awsSecretManager:
    podIdentity:
      provider: aws
    region: us-west-2
    secrets:
      - parameter: apiKey
        name: atlas-keda-secret
      - parameter: appKey
        name: atlas-keda-secret

ScaledObject

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: accounts-investor-accounts-api
  namespace: keda
spec:
  scaleTargetRef:
    name: accounts-investor-accounts-api
  maxReplicaCount: 5
  minReplicaCount: 1
  pollingInterval: 30
  cooldownPeriod: 60
  triggers:
    - type: datadog
      metricType: Value
      metadata:
        useClusterAgentProxy: "false"
        query: avg:kubernetes.cpu.usage.total{kube_namespace:keda, kube_deployment:accounts-investor-accounts-api}
        queryValue: "2000000"
        activationQueryValue: "1000000"
      authenticationRef:
        name: keda-clustertriggerauth
        kind: ClusterTriggerAuthentication

Anything else ?

We checked the DD SITE and it is datadoghq ("https://app.datadoghq.com/")
We checked whether there are any leading white spaces and trailing characters at secrets - None observed.
We are using secret from AWS secret manager. Confirmed that scale_handler can successfully pull and read the secrets from AWS.
We are configuring ClusterTriggerAuthentication CRD, so namespaced scope doesn't come into picture for secret as well as ClusterTriggerAuthentication object. Only for scaledObject namespace is used.
We referred most of the KB sources at Keda official site for Datadog scaler. Also referred previous github similar issues and discussions. Have sought help in other github discussions too.

senthilprasanth · 2024-12-08T00:26:00Z

senthilprasanth
Dec 8, 2024
Author

@JorTurFer : Your help is much appreciated.

0 replies

senthilprasanth · 2024-12-10T19:00:32Z

senthilprasanth
Dec 10, 2024
Author

Team, Looking for help. Your help is much valuable and appreciated

0 replies

JorTurFer · 2024-12-15T20:04:37Z

JorTurFer
Dec 15, 2024
Maintainer

Hello
Are both keys the same?

secrets:
      - parameter: apiKey
        name: atlas-keda-secret
      - parameter: appKey
        name: atlas-keda-secret

using this config, you are giving the same value to both parameters, and IIRC, they are different. @arapulido ?

0 replies

senthilprasanth · 2024-12-16T01:22:37Z

senthilprasanth
Dec 16, 2024
Author

@JorTurFer : Thanks for the reply !
I believe, my understanding in passing the parameters with right keys in the catch here.
I changed the approach and tested out and still receiving the 403 Forbidden error. Below is the approach.

Created 3 AWS secrets of type "Other type of secret" at AWS secrets manager
atlas-keda-secret to hold apiKey key-value pair { apiKey: XXX }
atlas-keda-secret1 to hold appKey key-value pair {appKey: XXX}
Create 1 ClusterTriggerAuthentication CR with following config

apiVersion: keda.sh/v1alpha1
kind: ClusterTriggerAuthentication
metadata:
  name: keda-clustertriggerauth
spec:
  awsSecretManager:
    podIdentity:
      provider: aws
    region: us-west-2
    secrets:
      - parameter: apiKey
        name: atlas-keda-secret
      - parameter: appKey
        name: atlas-keda-secret1

Created 1 ScaledObject CR with following config

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: accounts-investor-accounts-api
  namespace: keda
spec:
  scaleTargetRef:
    name: accounts-investor-accounts-api
  maxReplicaCount: 5
  minReplicaCount: 1
  pollingInterval: 30
  cooldownPeriod: 60
  triggers:
    - type: datadog
      metricType: Value
      metadata:
        useClusterAgentProxy: "false"
        query: avg:kubernetes.cpu.usage.total{kube_namespace:keda, kube_deployment:accounts-investor-accounts-api}
        queryValue: "2000000"
        activationQueryValue: "1000000"
      authenticationRef:
        name: keda-clustertriggerauth
        kind: ClusterTriggerAuthentication

Tested all the following combinations

DD apiKey & appKey was granted at full scope to sort out all permission related issues.
Stored these keys as plain text in AWS and tested. Result - 403 Forbidden
Stored these keys as base64 encoded and tested. Result - 403 Forbidden
Stored dataDogSite variable in another AWS secret atlas-keda-secret2 and tested - 403 Forbidden

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Keda-Datadog Scaler] Error establishing Datadog connection - 403 Forbidden connecting to Datadog API endpoint #6408

{{title}}

Replies: 4 comments

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

[Keda-Datadog Scaler] Error establishing Datadog connection - 403 Forbidden connecting to Datadog API endpoint #6408

senthilprasanth Dec 8, 2024

Report

KBs used for implementation

Expected Behavior

Actual Behavior

Steps to reproduce the problem

Logs from KEDA operator

KEDA version

Kubernetes Version

Platform

Scaler Details

cURL check

Config used

Anything else ?

Replies: 4 comments

senthilprasanth Dec 8, 2024 Author

senthilprasanth Dec 10, 2024 Author

JorTurFer Dec 15, 2024 Maintainer

senthilprasanth Dec 16, 2024 Author

senthilprasanth
Dec 8, 2024

senthilprasanth
Dec 8, 2024
Author

senthilprasanth
Dec 10, 2024
Author

JorTurFer
Dec 15, 2024
Maintainer

senthilprasanth
Dec 16, 2024
Author