autoscalingListener cannot get access token for Github App auth

[yijeong-cho]

I have deployed github runner following this guide
https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/deploying-runner-scale-sets-with-actions-runner-controller

I have created github app as guided,
and registered kubernetes secret as guided

kubectl create secret generic pre-defined-secret \
  --namespace=arc-runners \
  --from-literal=github_app_id=123456 \
  --from-literal=github_app_installation_id=654321 \
  --from-literal=github_app_private_key='-----BEGIN RSA PRIVATE KEY-----********'

and to my scaleset value, I added

  githubConfigSecret: gha-runner-scale-set-credential # pre-defined-k8s-secret

Therefore listener generated its own secret as below.

however, i still get this error....

2025-02-05T06:54:02Z	INFO	Starting metrics server	{"address": ":8080"}
2025-02-05T06:54:02Z	INFO	refreshing token	{"githubConfigUrl": "https://github.com/xxxxx"}
2025-02-05T06:54:02Z	INFO	getting access token for GitHub App auth	{"accessTokenURL": "https://api.github.com/app/installations/xxxxx/access_tokens"}
2025-02-05T06:54:02Z	ERROR	Retryable client error	{"error": "Post \"https://api.github.com/app/installations/xxxxx/access_tokens\": dial tcp 20.200.245.245:443: connect: connection refused", "method": "POST", "url": "https://api.github.com/app/installations/xxxxx/access_tokens", "error": "request failed"}
github.com/actions/actions-runner-controller/github/actions.(*clientLogger).Error
	github.com/actions/actions-runner-controller/github/actions/client.go:77
github.com/hashicorp/go-retryablehttp.(*Client).Do
	github.com/hashicorp/[email protected]/client.go:718
github.com/hashicorp/go-retryablehttp.(*RoundTripper).RoundTrip
	github.com/hashicorp/[email protected]/roundtripper.go:47
net/http.send
	net/http/client.go:259
net/http.(*Client).send
	net/http/client.go:180
net/http.(*Client).do
	net/http/client.go:724
net/http.(*Client).Do
	net/http/client.go:590
github.com/actions/actions-runner-controller/github/actions.(*Client).Do
	github.com/actions/actions-runner-controller/github/actions/client.go:274
github.com/actions/actions-runner-controller/github/actions.(*Client).fetchAccessToken
	github.com/actions/actions-runner-controller/github/actions/client.go:1052
github.com/actions/actions-runner-controller/github/actions.(*Client).getRunnerRegistrationToken
	github.com/actions/actions-runner-controller/github/actions/client.go:986
github.com/actions/actions-runner-controller/github/actions.(*Client).updateTokenIfNeeded
	github.com/actions/actions-runner-controller/github/actions/client.go:1256
github.com/actions/actions-runner-controller/github/actions.(*Client).NewActionsServiceRequest
	github.com/actions/actions-runner-controller/github/actions/client.go:306
github.com/actions/actions-runner-controller/github/actions.(*Client).doSessionRequest
	github.com/actions/actions-runner-controller/github/actions/client.go:699
github.com/actions/actions-runner-controller/github/actions.(*Client).CreateMessageSession
	github.com/actions/actions-runner-controller/github/actions/client.go:681
main.createSession
	github.com/actions/actions-runner-controller/cmd/githubrunnerscalesetlistener/autoScalerMessageListener.go:71
main.NewAutoScalerClient
	github.com/actions/actions-runner-controller/cmd/githubrunnerscalesetlistener/autoScalerMessageListener.go:45
main.run
	github.com/actions/actions-runner-controller/cmd/githubrunnerscalesetlistener/main.go:187
main.main.func1
	github.com/actions/actions-runner-controller/cmd/githubrunnerscalesetlistener/main.go:83
golang.org/x/sync/errgroup.(*Group).Go.func1
	golang.org/x/[email protected]/errgroup/errgroup.go:78
2025-02-05T06:54:03Z	INFO	getting runner registration token	{"registrationTokenURL": "https://api.github.com/orgs/xxxxx/actions/runners/registration-token"}
2025-02-05T06:54:03Z	INFO	getting Actions tenant URL and JWT	{"registrationURL": "https://api.github.com/actions/runner-registration"}
2025-02-05T06:54:04Z	INFO	auto_scaler	current runner scale set statistics.	{"statistics": "{\"totalAvailableJobs\":0,\"totalAcquiredJobs\":0,\"totalAssignedJobs\":0,\"totalRunningJobs\":0,\"totalRegisteredRunners\":1,\"totalBusyRunners\":0,\"totalIdleRunners\":0}"}
2025-02-05T06:54:04Z	INFO	service	waiting for message...
2025-02-05T06:54:04Z	INFO	service	process message.	{"messageId": 0, "messageType": "RunnerScaleSetJobMessages"}
2025-02-05T06:54:04Z	INFO	service	current runner scale set statistics.	{"available jobs": 0, "acquired jobs": 0, "assigned jobs": 0, "running jobs": 0, "registered runners": 1, "busy runners": 0, "idle runners": 0}
2025-02-05T06:54:04Z	INFO	service	try scale runner request up/down base on assigned job count	{"assigned job": 0, "decision": 1, "min": 1, "max": 5, "currentRunnerCount": -1}
2025-02-05T06:54:04Z	INFO	KubernetesManager	Created merge patch json for EphemeralRunnerSet update	{"json": "{\"spec\":{\"replicas\":1}}"}
2025-02-05T06:54:04Z	INFO	KubernetesManager	Ephemeral runner set scaled.	{"namespace": "arc-system", "name": "tig-runner-focal-x64-cqvmq", "replicas": 1}
2025-02-05T06:54:04Z	INFO	service	waiting for message...

I confirmed that there is no issue with network.
when i run curl command in the container,
i get as below.

istio-proxy@tig-runner-focal-x64-7948666d-listener:/$ curl -v https://api.github.com/app/installations/60453291/access_tokens
* Host api.github.com:443 was resolved.
* IPv6: (none)
* IPv4: 20.200.245.245
*   Trying 20.200.245.245:443...
* Connected to api.github.com (20.200.245.245) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / X25519 / id-ecPublicKey
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), signed using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), signed using ecdsa-with-SHA384
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://api.github.com/app/installations/60453291/access_tokens
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: api.github.com]
* [HTTP/2] [1] [:path: /app/installations/60453291/access_tokens]
* [HTTP/2] [1] [user-agent: curl/8.5.0]
* [HTTP/2] [1] [accept: */*]
> GET /app/installations/60453291/access_tokens HTTP/2
> Host: api.github.com
> User-Agent: curl/8.5.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 401 
< date: Wed, 05 Feb 2025 07:27:06 GMT
< content-type: application/json; charset=utf-8
< content-length: 131
< x-github-media-type: github.v3; format=json
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< vary: Accept-Encoding, Accept, X-Requested-With
< server: github.com
< x-ratelimit-limit: 60
< x-ratelimit-remaining: 57
< x-ratelimit-reset: 1738743320
< x-ratelimit-resource: core
< x-ratelimit-used: 3
< x-github-request-id: 9E00:0A6E:1E0A29:27D491:67A312CA
< 
{
  "message": "A JSON web token could not be decoded",
  "documentation_url": "https://docs.github.com/rest",
  "status": "401"
}
* Connection #0 to host api.github.com left intact

also, i actually can get the token inside the pod (istio sidecar container) ...

istio-proxy@tig-runner-focal-x64-7948666d-listener:/$ curl --request POST \
--url "https://api.github.com/app/installations/60453291/access_tokens" \
--header "Accept: application/vnd.github+json" \
--header "Authorization: Bearer ewogICAgInR5cCI6IkpXVCIsCiAgICAiYWxnIjoiUlMyNTYiCn0.....xxxxxx" \
--header "X-GitHub-Api-Version: 2022-11-28"
{
  "token": "ghs_xxxxxxxxxxxxx",
  "expires_at": "2025-02-05T10:24:10Z",
  "permissions": {
    "organization_self_hosted_runners": "write",
    "administration": "write",
    "metadata": "read"
  },
  "repository_selection": "all"
}
istio-proxy@tig-runner-focal-x64-7948666d-listener:/$ 

Can someone help me? ;(