Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[PLugin] OpenTelemetry Tracer Plugin make request processing time increase #9961

Open
HugManh opened this issue Jul 6, 2023 · 7 comments · May be fixed by #11873
Open

[PLugin] OpenTelemetry Tracer Plugin make request processing time increase #9961

HugManh opened this issue Jul 6, 2023 · 7 comments · May be fixed by #11873
Assignees
Labels

Comments

@HugManh
Copy link

HugManh commented Jul 6, 2023

Hi everyone,
I using TrafficSrever branch: master - #828b7d3. I trying OpenTelemetry Tracer Plugin to manage tracer for system. When i use the plugin, it make request processing time increase.

When use plugin:
has_otel
req_has_otel

Don't use plugin:
no_otel
req_no_otel

Has anyone used this plugin yet?

@shukitchan shukitchan self-assigned this Jul 6, 2023
@shukitchan
Copy link
Contributor

How does your observability stack look like?

The plugin is sending a OTLP http request for sending the trace info to a backend. So if your backend is quite far away, it can take quite some time.

One alternative is this - https://opentelemetry.io/docs/collector/deployment/agent/
You can have the deployment of an agent/collector on the same host of your ATS. And you configure the plugin to send OTLP request to the agent/collector instead. That should be much faster. And then you let the agent/collector do its work to send the info to a real backend.

@HugManh
Copy link
Author

HugManh commented Jul 6, 2023

How does your observability stack look like?

The plugin is sending a OTLP http request for sending the trace info to a backend. So if your backend is quite far away, it can take quite some time.

My backend is pretty close so I think it's not due to this. I tried with opentelemetry-cpp version 1.3.0 and as a result request time is faster than branch master . Note: The results I posted above is from the opentelemetry branch master#.
Do you think the problem is with the sdk?
Result of version 1.3.0:
photo_2023-07-06_22-00-08

One alternative is this - https://opentelemetry.io/docs/collector/deployment/agent/ You can have the deployment of an agent/collector on the same host of your ATS. And you configure the plugin to send OTLP request to the agent/collector instead. That should be much faster. And then you let the agent/collector do its work to send the info to a real backend.

I will try this later,
Thankyou @shukitchan

@shukitchan
Copy link
Contributor

300+ms is still a lot.
I think we should try having the agent/collector in the local host.

I have not got the time to try the latest opentelemetry-cpp release. So I can't say if there is anything changed in the sdk that will cause this much latency.

@HugManh
Copy link
Author

HugManh commented Jul 7, 2023

@shukitchan Have you tried using opentelemetry without collector?

@HugManh
Copy link
Author

HugManh commented Jul 7, 2023

I think i found problem in the plugin. I change span processor from SimpleSpanProcessor to BatchSpanProcessor. This solved my problem.
@shukitchan Thank you very much!

@shukitchan
Copy link
Contributor

What would be the number after the change?

Do you want to submit a pull request for the change?

Copy link

This issue has been automatically marked as stale because it has not had recent activity. Marking it stale to flag it for further consideration by the community.

@github-actions github-actions bot added the Stale label Jul 10, 2024
@shukitchan shukitchan removed the Stale label Jul 10, 2024
@shukitchan shukitchan linked a pull request Nov 17, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants