[chore][otel] Enable persistence in otel.yml #5549

VihasMakwana · 2024-09-17T11:53:49Z

Enhancement
Docs

What does this PR do?

This PR adds the filestorage extension in otel.yaml and updates the documentation.
Updates otel dependencies to 0.110.0

Why is it important?

By default, the OpenTelemetry Collector operates in a stateless mode, meaning it does not save file offsets to disk. To resolve this, we enable persistence in the default configuration files.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

How to test this PR locally

Build from this branch
Keep the config default and run elastic-agent otel --config otel.yaml
Ingest some data and stop the process.
Restart the process and ingest some more data.
You shouldn't see any data loss/duplication.

Related issues

Closes https://github.com/elastic/opentelemetry-dev/issues/281

mergify · 2024-09-17T11:54:29Z

This pull request does not have a backport label. Could you fix it @VihasMakwana? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-./d./d is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2024-09-17T11:54:30Z

backport-v8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

elasticmachine · 2024-09-18T10:19:53Z

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)

VihasMakwana · 2024-09-18T17:17:36Z

@elastic/elastic-agent-control-plane

I propose two options for default directory:

Set it as per current sample configurations and modify them during the onboarding process. Currently, the sample configs use STORAGE_DIR env as the default directory option
Follow Filebeat's directory layout and set the default to a path under path.data.

For the latter option, we need to implement a config converter which maps path.data to /opt/Elastic/Agent/data (in case of Linux).

Let me know how do you feel about this proposal.

cc: @ycombinator @pierrehilbert @andrzej-stencel

VihasMakwana · 2024-09-18T17:19:53Z

Introducing a new config converter would be straightforward and could be easily integrated into our distribution.

cmacknz · 2024-09-19T13:32:15Z

As discussed let's leave this as an environment variable. How the collector is run and installed (if it is installed as a service at all) is currently completely up to the user and this goes along with that.

The only place we likely want a default is in our reference Kubernetes configurations, which can set the environment variable or set a default in the configuration it includes directly.

In general we do not want to use any of the paths that are also used by Beats or Agent, because a standalone OTel collector must be able to run alongside them without issue.

In our OTel distribution I think we solve this by having an install command that puts this in whatever base install directory we pick (/opt/Elastic, C:\Program Files\Elastic, or /Library/Elastic) but we need an installer that runs as a privileged user to do that. You can't do it with config alone.

mergify · 2024-09-27T11:05:00Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

otel.yml

internal/pkg/otel/README.md

changelog/fragments/1726572104-enable-persistence-by-default.yaml

go.mod

andrzej-stencel · 2024-09-27T12:46:44Z

otel.yml

+  file_storage/filelogreceiver:
+    create_directory: true
+    directory: ${env:STORAGE_DIR}


The addition of the File Storage extension may cause problems for users using this configuration with read-only filesystems, like in Docker, in Kubernetes, unless they configure a writable storage layer in their containers/pods. Should we be concerned about this?

The addition of the File Storage extension may cause problems for users using this configuration with read-only filesystems, like in Docker, in Kubernetes, unless they configure a writable storage layer in their containers/pods. Should we be concerned about this?

I believe we will have a separate recommendation in our k8s configs based on @cmacknz's comment.

This change breaks running Elatic Agent as OTel Collector in Docker. I'd rather have this change applied in a way that doesn't break running the EDOT Collector in Docker. If we want to enable persistence by default, it should "just work", like it "just works" with the regular Elastic Agent.

$ docker run --rm -ti --entrypoint="elastic-agent" docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT otel Starting in otel mode 2024-09-30T09:06:39.260Z info [email protected]/service.go:137 Setting up own telemetry... 2024-09-30T09:06:39.261Z info [email protected]/service.go:186 Skipped telemetry setup. 2024-09-30T09:06:39.261Z info builders/builders.go:26 Development component. May change in the future. {"kind": "exporter", "data_type": "logs", "name": "debug"} 2024-09-30T09:06:39.262Z info builders/extension.go:50 Development component. May change in the future. {"kind": "extension", "name": "memory_limiter"} 2024-09-30T09:06:39.262Z info memorylimiter/memorylimiter.go:75 Memory limiter configured {"kind": "extension", "name": "memory_limiter", "limit_mib": 700, "spike_limit_mib": 180, "check_interval": 1} 2024-09-30T09:06:39.252Z warn [email protected]/provider.go:59 Configuration references unset environment variable {"name": "STORAGE_DIR"} failed to build extensions: failed to create extension "file_storage/filelogreceiver": mkdir /var/lib/otelcol: permission denied

@andrzej-stencel I see what you mean. The error is due to an empty env:STORAGE_DIR and hence, the filestorage extension defaults to /var/lib/otelcol. Since the default user in the Docker container is elastic-agent, this causes a permission denied error.

If I use any of following commands, it works.

docker run --rm -ti --entrypoint="elastic-agent" --mount type=bind,source=$(pwd)/data,target=/usr/share/otel -e STORAGE_DIR=/usr/share/otel/ docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT otel

or

docker run --user root --rm -ti --entrypoint="elastic-agent" docker.elastic.co/beats/elastic-agent:9.0.0-SNAPSHOT otel

I wonder how should we approach this. Maybe add documentation? or update EDOT according to @cmacknz's comment. Or do both.

In our OTel distribution I think we solve this by having an install command that puts this in whatever base install directory we pick (/opt/Elastic, C:\Program Files\Elastic, or /Library/Elastic) but we need an installer that runs as a privileged user to do that. You can't do it with config alone.

@cmacknz @andrzej-stencel I'd appreciate your opinions over this!

I agree with you that is should "just work".

In the container we control the paths, and can pick a default that is writable by our user in the container. Elastic agent uses the STATE_PATH environment variable and puts it under /usr/share/elastic-agent.

elastic-agent/internal/pkg/agent/cmd/container.go

Lines 48 to 49 in 95021e1

agentBaseDirectory = "/usr/share/elastic-agent" // directory that holds all elastic-agent related files

defaultStateDirectory = agentBaseDirectory + "/state" // directory that will hold the state data

We could add an /otelcol sub-directory to /usr/share/elastic-agent for one possibility.

Possibly we can reuse the STATE_PATH variable and change which sub-directory we use based on whether the entrypoint is the otel command or the elastic-agent container command.

Makes sense.

Co-authored-by: Andrzej Stencel <[email protected]>

mergify · 2024-10-07T07:04:42Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

VihasMakwana · 2024-10-08T15:20:21Z

@andrzej-stencel @cmacknz PTAL 😅

andrzej-stencel

I'm not fond of the prepareEnv function setting the STORAGE_DIR env var irrespective of whether the env var is used in configuration or not. On the other hand, I don't have an alternative "perfect" solution, so let's go with this approach now and perhaps change if we come up with a better way.

mergify · 2024-10-11T16:11:16Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

internal/pkg/otel/templates/README.md.tmpl

cmacknz · 2024-10-11T19:53:11Z

internal/pkg/agent/cmd/otel.go

+		// The filestorage extension will handle directory creation since create_directory: true is set by default.
+		// If the user hasn’t specified the env:STORAGE_DIR in filestorage, they may have opted for a custom path, and the extension will create the directory accordingly.
+		// In this case, setting env:STORAGE_DIR will have no effect.
+		if err := os.Setenv("STORAGE_DIR", filepath.Join(paths.Top(), "otel_registry")); err != nil {


nit: I would just call the path otel instead of otel_registry since there is no guarantee we only use this directory to store file log registries long term.

cmacknz · 2024-10-11T19:55:39Z

internal/pkg/agent/cmd/otel.go

+		// The filestorage extension will handle directory creation since create_directory: true is set by default.
+		// If the user hasn’t specified the env:STORAGE_DIR in filestorage, they may have opted for a custom path, and the extension will create the directory accordingly.
+		// In this case, setting env:STORAGE_DIR will have no effect.
+		if err := os.Setenv("STORAGE_DIR", filepath.Join(paths.Top(), "otel_registry")); err != nil {


Also, why can't we just reuse the STATE_PATH environment variable the container already supports so we don't need to add a new one? The default value is also writable by the elastic agent container user.

One reason to do this is it won't be a breaking change if we decide to get rid of STORAGE_DIR later.

Co-authored-by: Craig MacKenzie <[email protected]>

VihasMakwana · 2024-10-11T22:08:32Z

@cmacknz I've reused the STATE_PATH and changed directory name. PTAL!

cmacknz · 2024-10-15T16:46:29Z

internal/pkg/agent/cmd/otel.go

@@ -118,3 +123,22 @@ func runCollector(cmdCtx context.Context, configFiles []string) error {
 	return goerrors.Join(errs...)

 }
+
+func prepareEnv() error {
+	if _, ok := os.LookupEnv("STORAGE_DIR"); !ok {


Can you use STATE_PATH here as well (and in the collector examples)?

What I am trying to do is avoid introducing a new environment variable we may decide to remove in the future, trying to keep our public API as small as possible while we figure out how everything fits together.

STATE_PATH isn't going anywhere without making it a breaking change for many people, so it doesn't increase the surface area of configuration we need to maintain.

mergify · 2024-10-15T19:23:51Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b enabl-file-storage-update-docs upstream/enabl-file-storage-update-docs
git merge upstream/main
git push upstream enabl-file-storage-update-docs

cmacknz

Thanks!

elastic-sonarqube · 2024-10-21T08:35:18Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

* chore: update config, docs * fix: spelling mistake * fix: go tidy * remove comments * fix: go tidy * fix: add filestorage to extension * Update changelog/fragments/1726572104-enable-persistence-by-default.yaml Co-authored-by: Andrzej Stencel <[email protected]> * chore: update readme * chore: update readme * go.mod * chore: go.mod * comments * comments * chore: update notice * chore: go.sum and notice * chore: go.sum and notice * Update internal/pkg/otel/templates/README.md.tmpl Co-authored-by: Craig MacKenzie <[email protected]> * Update internal/pkg/otel/templates/README.md.tmpl Co-authored-by: Craig MacKenzie <[email protected]> * chore: address comments * chore: use STATE_PATH * chore: update readme * fix: add missing import --------- Co-authored-by: Andrzej Stencel <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]> (cherry picked from commit 49579ae)

VihasMakwana added 2 commits September 17, 2024 17:12

chore: update config, docs

2d56cba

Merge branch 'main' into enabl-file-storage-update-docs

038027b

VihasMakwana added documentation Improvements or additions to documentation enhancement New feature or request Team:Elastic-Agent-Control-Plane Label for the Agent Control Plane team opentelemetry Related to the Elastic Distribution of the OpenTelemetry Collector labels Sep 17, 2024

VihasMakwana self-assigned this Sep 17, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 17, 2024

VihasMakwana and others added 2 commits September 17, 2024 18:12

fix: spelling mistake

a82d734

Merge branch 'main' into enabl-file-storage-update-docs

c28889f

VihasMakwana marked this pull request as ready for review September 18, 2024 10:19

VihasMakwana requested a review from a team as a code owner September 18, 2024 10:19

VihasMakwana requested review from blakerouse and pchila September 18, 2024 10:19

VihasMakwana requested a review from andrzej-stencel September 18, 2024 10:20

pchila approved these changes Sep 24, 2024

View reviewed changes

VihasMakwana added 2 commits September 27, 2024 16:34

fix: go tidy

539931e

remove comments

564a619

VihasMakwana added 2 commits September 27, 2024 16:37

Merge branch 'main' into enabl-file-storage-update-docs

36238a3

fix: go tidy

a5cbe5f

andrzej-stencel reviewed Sep 27, 2024

View reviewed changes

VihasMakwana and others added 2 commits September 29, 2024 16:38

fix: add filestorage to extension

899607a

Update changelog/fragments/1726572104-enable-persistence-by-default.yaml

eafd9cd

Co-authored-by: Andrzej Stencel <[email protected]>

VihasMakwana and others added 2 commits October 4, 2024 18:29

chore: update notice

1f0ecc9

Merge branch 'main' into enabl-file-storage-update-docs

b17e355

VihasMakwana added 3 commits October 8, 2024 20:45

Merge branch 'main' into enabl-file-storage-update-docs

b203fb4

chore: go.sum and notice

eac340c

chore: go.sum and notice

5baa9ac

andrzej-stencel approved these changes Oct 11, 2024

View reviewed changes

cmacknz reviewed Oct 11, 2024

View reviewed changes

VihasMakwana and others added 3 commits October 12, 2024 02:46

Update internal/pkg/otel/templates/README.md.tmpl

55349ef

Co-authored-by: Craig MacKenzie <[email protected]>

Update internal/pkg/otel/templates/README.md.tmpl

6be319d

Co-authored-by: Craig MacKenzie <[email protected]>

chore: address comments

18e0231

VihasMakwana requested a review from cmacknz October 11, 2024 22:08

cmacknz reviewed Oct 15, 2024

View reviewed changes

VihasMakwana and others added 3 commits October 16, 2024 02:37

chore: use STATE_PATH

315eadb

chore: update readme

aa76e2f

Merge branch 'main' into enabl-file-storage-update-docs

2387b51

VihasMakwana requested a review from cmacknz October 15, 2024 21:12

cmacknz approved these changes Oct 16, 2024

View reviewed changes

fix: add missing import

75b4147

pchila removed their request for review October 18, 2024 12:56

Merge branch 'main' into enabl-file-storage-update-docs

03b5901

VihasMakwana merged commit 49579ae into main Oct 21, 2024
14 checks passed

VihasMakwana deleted the enabl-file-storage-update-docs branch October 21, 2024 10:43

mergify bot mentioned this pull request Oct 21, 2024

[8.x](backport #5549) [chore][otel] Enable persistence in otel.yml #5830

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[chore][otel] Enable persistence in otel.yml #5549

[chore][otel] Enable persistence in otel.yml #5549

VihasMakwana commented Sep 17, 2024 •

edited

Loading

mergify bot commented Sep 17, 2024

mergify bot commented Sep 17, 2024

elasticmachine commented Sep 18, 2024

VihasMakwana commented Sep 18, 2024 •

edited

Loading

VihasMakwana commented Sep 18, 2024

cmacknz commented Sep 19, 2024

mergify bot commented Sep 27, 2024

andrzej-stencel Sep 27, 2024

VihasMakwana Sep 29, 2024

andrzej-stencel Sep 30, 2024 •

edited

Loading

VihasMakwana Sep 30, 2024

VihasMakwana Sep 30, 2024

cmacknz Oct 2, 2024 •

edited

Loading

cmacknz Oct 2, 2024

VihasMakwana Oct 3, 2024

mergify bot commented Oct 7, 2024

VihasMakwana commented Oct 8, 2024

andrzej-stencel left a comment

mergify bot commented Oct 11, 2024

cmacknz Oct 11, 2024

cmacknz Oct 11, 2024

VihasMakwana commented Oct 11, 2024

cmacknz Oct 15, 2024 •

edited

Loading

mergify bot commented Oct 15, 2024

cmacknz left a comment

elastic-sonarqube bot commented Oct 21, 2024

	agentBaseDirectory = "/usr/share/elastic-agent" // directory that holds all elastic-agent related files
	defaultStateDirectory = agentBaseDirectory + "/state" // directory that will hold the state data

[chore][otel] Enable persistence in otel.yml #5549

[chore][otel] Enable persistence in otel.yml #5549

Conversation

VihasMakwana commented Sep 17, 2024 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

mergify bot commented Sep 17, 2024

mergify bot commented Sep 17, 2024

elasticmachine commented Sep 18, 2024

VihasMakwana commented Sep 18, 2024 • edited Loading

VihasMakwana commented Sep 18, 2024

cmacknz commented Sep 19, 2024

mergify bot commented Sep 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrzej-stencel Sep 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cmacknz Oct 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mergify bot commented Oct 7, 2024

VihasMakwana commented Oct 8, 2024

andrzej-stencel left a comment

Choose a reason for hiding this comment

mergify bot commented Oct 11, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

VihasMakwana commented Oct 11, 2024

cmacknz Oct 15, 2024 • edited Loading

Choose a reason for hiding this comment

mergify bot commented Oct 15, 2024

cmacknz left a comment

Choose a reason for hiding this comment

elastic-sonarqube bot commented Oct 21, 2024

Quality Gate passed

VihasMakwana commented Sep 17, 2024 •

edited

Loading

VihasMakwana commented Sep 18, 2024 •

edited

Loading

andrzej-stencel Sep 30, 2024 •

edited

Loading

cmacknz Oct 2, 2024 •

edited

Loading

cmacknz Oct 15, 2024 •

edited

Loading