Add ability to enroll with a specific ID

[blakerouse]

What is the problem this PR solves?

This solves an issue where an Elastic Agent is being replaced with a new Elastic Agent instance for the same host, pod, or workload. This allows the enrolling Elastic Agent to tell the ID that it wants to use, that ID can be currently in-use and this enrollment will take over the record of that Elastic Agent. To take off the existing Elastic Agent both the original and the new enrollment must use the same replace_token during the enrollment. This ensures that the original enrollment informs Fleet Server that it can be replaced, and ensures that the replacement has the same token to perform the replacement.

How does this PR solve the problem?

It solves the issue by taking a new id field in the enroll HTTP request. That id is then used as the Elastic Agent ID and determines if this is a new Elastic Agent or if it should take over an existing Elastic Agent record.

How to test this PR locally

At the moment the integration tests are the best way to test this, as the ability to use this field has not been exposed yet on the Elastic Agent.

Design Checklist

• I have ensured my design is stateless and will work when multiple fleet-server instances are behind a load balancer.
• I have or intend to scale test my changes, ensuring it will work reliably with 100K+ agents connected.
• I have included fail safe mechanisms to limit the load on fleet-server: rate limiting, circuit breakers, caching, load shedding, etc. (already covered by enroll handle)

Checklist

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files (no config changes)
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool

Related issues

• Closes Add ability to provide the agent-id in enroll API #4226

[jlind23]

After chatting with @blakerouse over slack we should fail the enrollment of this "new" agent if the policy has temper protection enabled.

[kaanyalti]

Changes so far look good to me, waiting for updates relevant to @jlind23's comment above to approve

[michel-laterman]

This is a pretty big change from our discussion.
Please also add a sentence saying the (replaced) agent will still be able to send data into ES

[blakerouse]

It is a slight change, because it is not possible to get an API key token again after the initial create. That made me have no choice to change the behavior.

[blakerouse]

I did add more as requested to the description of this field to inform that it will still allow data to flow.

[jlind23]

Changes so far look good to me, waiting for updates relevant to @jlind23's comment above to approve

See #4226 (comment)

[blakerouse]

@jlind23 @michel-laterman @kaanyalti I have updated this PR based on the discuss I had with @jlind23 about security with this feature. This PR now includes an additional replace_token during the enrollment API. I have updated the PR description to describe this as well as the API specification describes it.

[pkoutsovasilis]

Except a small ending of a trace-span the code changes look good to me. I understand the potential pitfalls with this feature and definitely see how the replace_token helps in minimising some of them but still this feature to me serves only special-case scenarios and is not streamlined usage 🙂

[blakerouse]

@pkoutsovasilis @michalpristas I updated the PR with the request fixes. Thanks for the reviews.

[elastic-sonarqube]

Quality Gate passed

Issues
2 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
87.3% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[pkoutsovasilis]

LGTM

[michalpristas]

Thanks for resolving comment. one question about behavior other than that I'm ok with the change.
i let you decide how you want to address the point i raised in this iteration

[michalpristas]

can it return agent.id == ""? if not this if statement is not needed.
if so. we dont have this case handled as this should not be the same as empty ID when req.ID is not used.
we whould probably not continue with empty id, probably we should fail. generating a new one breaks the purpose of providing it via req.id

[blakerouse]

It absolutely will return an agent.id == "". This happens when we check to see if an existing agent already exists with that ID. The ErrNotFound will not be returned from _checkAgent, it will return nil error and this will be `agent.id == "".