[MAC15]: Agent doesn't come up on Fleet UI on reboot after running install command with --delay-enroll.

[amolnater-qasource]

Kibana Build details:

VERSION: 8.15.1 
BUILD: 76534
COMMIT: f66ec5b0ddd990d103489c47ca1bcb97dc50bc6b

Artifact: https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.15.1-darwin-x86_64.tar.gz

Host: MAC15

Preconditions:

1. 8.15.1 kibana cloud environment should be available.

Steps to reproduce:

1. Run agent install command with --delay-enroll.
2. Restart the mac.
3. Observe no Agent doesn't come up on Fleet UI.

NOTE:

• Agent gets installed with delay enroll on mac version: 14.7

Expected Result:
Agent should come up on Fleet UI on reboot after running install command with --delay-enroll.

Agent Directory:
curl -L -H 'Authorization: b480faf457fc2ab0' -o 'agent.zip' https://upload.elastic.co/d/fe08154ec7ce49a60f17ce26f5cceae7b372253a3430d8266c879b535cacf222

Related Ticket:
https://github.com/elastic/ingest-dev/issues/4032

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

[amolnater-qasource]

@muskangulati-qasource Please review.

[muskangulati-qasource]

Secondary review on this ticket is Done!

[cmacknz]

The logs say agent is trying to enroll but can't because the token is not valid UTF-8 text: fleet-server: status code: 401, fleet-server returned an error: ErrInvalidToken, message: token not valid utf8.

My first suspicion is the token wasn't provided to the agent correctly (missing some part of it perhaps, the trailing == padding is sometimes missed). Is this reproducible?

{"log.level":"warn","@timestamp":"2024-09-19T07:26:59.413Z","log.logger":"transport","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3","file.name":"transport/tcp.go","file.line":53},"message":"DNS lookup failure \"af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com\": lookup af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com: no such host","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:26:59.413Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":524},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/ with exponential backoff (init 1s, max 10s)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T07:26:59.413Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.tryDelayEnroll","file.name":"cmd/run.go","file.line":583},"message":"failed to perform delayed enrollment (will try again): fail to enroll: fail to execute request to fleet-server: lookup af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com: no such host","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:26:59.678Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2024-09-19T07:26:59.881Z","log.logger":"transport","log.origin":{"function":"github.com/elastic/elastic-agent-libs/transport/httpcommon.(*HTTPTransportSettings).RoundTripper.NetDialer.TestNetDialer.func3","file.name":"transport/tcp.go","file.line":53},"message":"DNS lookup failure \"af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com\": lookup af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com: no such host","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:26:59.881Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":524},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/ with exponential backoff (init 1s, max 10s)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T07:26:59.881Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.tryDelayEnroll","file.name":"cmd/run.go","file.line":583},"message":"failed to perform delayed enrollment (will try again): fail to enroll: fail to execute request to fleet-server: lookup af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com: no such host","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:00.555Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:00.906Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":524},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/ with exponential backoff (init 1s, max 10s)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T07:27:00.906Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.tryDelayEnroll","file.name":"cmd/run.go","file.line":583},"message":"failed to perform delayed enrollment (will try again): fail to enroll: fail to execute request to fleet-server: dial tcp 35.235.72.223:443: connect: network is unreachable","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:01.463Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:01.668Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":524},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/ with exponential backoff (init 1s, max 10s)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T07:27:01.668Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.tryDelayEnroll","file.name":"cmd/run.go","file.line":583},"message":"failed to perform delayed enrollment (will try again): fail to enroll: fail to execute request to fleet-server: dial tcp 35.235.72.223:443: connect: network is unreachable","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:01.937Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:02.976Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":524},"message":"1st enrollment attempt failed, retrying enrolling to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/ with exponential backoff (init 1s, max 10s)","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-09-19T07:27:02.976Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.tryDelayEnroll","file.name":"cmd/run.go","file.line":583},"message":"failed to perform delayed enrollment (will try again): fail to enroll: fail to execute request to fleet-server: status code: 401, fleet-server returned an error: ErrInvalidToken, message: token not valid utf8","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2024-09-19T07:27:03.634Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/cmd.(*enrollCmd).enrollWithBackoff","file.name":"cmd/enroll_cmd.go","file.line":518},"message":"Starting enrollment to URL: https://af15cb7cb15d4a06bfbeb5b6426134e6.fleet.us-west2.gcp.elastic-cloud.com:443/","log":{"source":"elastic-agent"},"ecs.version":"1.6.0"}

[amolnater-qasource]

Hi @cmacknz

Thank you for looking into this issue.
We had revalidated this issue on fresh 8.15.1 kibana cloud environment and found it working fine now.

Observations:

• Agent comes up on Fleet UI on reboot after running install command with --delay-enroll.

Screenshot:

Hence we are closing and marking this issue as QA:Validated.
Thanks!