fix(celery): close `celery.apply` spans even without after_task_publish, when using apply_async #10676

wantsui · 2024-09-16T21:13:24Z

The instrumentation for the Celery integration relies on various Celery signals in order to start and end the span when calling on apply_async.

The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces).

Example:

dd-trace-py expects the signal before_task_publish to start the span then after_task_publish to close the span. If the after_task_publish signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish.
The same thing above can also happen to task_prerun and task_postrun.

Solution

This PR patches apply_async so that there is a check to see if there is a span lingering around and closes it when apply_task is called.

If an internal exception happens, the error will be marked on the celery.apply span.

To track this, I added new logs in debug mode:

The after_task_publish signal was not called, so manually closing span

and

The task_postrun signal was not called, so manually closing span

There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not.

Special Thanks:

Thanks to @tabgok for going through this with me in great detail!
@timmc-edx for helping us track it down!

APMS-13158

Checklist

PR author has checked that all the criteria below are met
The PR description includes an overview of the change
The PR description articulates the motivation for the change
The change includes tests OR the PR description describes a testing strategy
The PR description notes risks associated with the change, if any
Newly-added code is easy to change
The change follows the library release note guidelines
The change includes or references documentation updates if necessary
Backport labels are set (if applicable)

Reviewer Checklist

Reviewer has checked that all the criteria below are met
Title is accurate
All changes are related to the pull request's stated goal
Avoids breaking API changes
Testing strategy adequately addresses listed risks
Newly-added code is easy to change
Release note makes sense to a user of the library
If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
Backport labels are set in a manner that is consistent with the release branch maintenance policy

APMS-13158

…signal does not get called

datadog-dd-trace-py-rkomorn · 2024-09-16T21:49:35Z

Datadog Report

Branch report: close-apply-async-celery-spans
Commit report: 81b2eb9
Test service: dd-trace-py

✅ 0 Failed, 820 Passed, 376 Skipped, 21m 13.4s Total duration (15m 15.32s time saved)

github-actions · 2024-09-16T21:52:03Z

CODEOWNERS have been resolved as:

releasenotes/notes/fix-celery-apply-async-span-close-b7a8db188459f5b5.yaml  @DataDog/apm-python
ddtrace/contrib/internal/celery/app.py                                  @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/celery/signals.py                              @DataDog/apm-core-python @DataDog/apm-idm-python
tests/contrib/celery/test_integration.py                                @DataDog/apm-core-python @DataDog/apm-idm-python

pr-commenter · 2024-09-16T22:11:02Z

Benchmarks

Benchmark execution time: 2024-10-01 20:02:33

Comparing candidate commit 81b2eb9 in PR branch close-apply-async-celery-spans with baseline commit 7f6f3db in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 371 metrics, 53 unstable metrics.

…dingly.

…he after task publish signal is not called, there are no spans at the moment so the test fails.

tests/contrib/celery/test_integration.py

update with latest changes on the main branch

add changes from main branch

@timmc-edx

… 2.14] (#10874) Backport 6346fcb from #10848 to 2.14. This PR fixes an issue where Celery's closing signals got triggered but dd-trace-py skipped closing the `celery.apply` span due to not finding the task id. In celery's `task_protocol: 1`, the id is in the message of the body: https://docs.celeryq.dev/en/main/internals/protocol.html#message-body . The issue with the previous logic is that if the headers does have information (even if the headers were unrelated to the id), it would skip the check of the id in the body: before: ``` if headers: ``` after (this PR): ``` if headers and 'id' in headers: ``` By doing this, we check the headers for the id, then check the body for the id. If it fails to find the task id in the body or header, then it still hits the debug log, `unable to extract the Task and the task_id. This version of Celery may not be supported.` . This PR relates to the goal of #10676 , to close celery spans. If for some reason the logic in this PR fails to close an open `celery.apply` span, #10676 will act as a fail safe and close it. Special Thanks: @timmc-edx for helping us track this down! ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: wantsui <[email protected]>

@tabgok

…sh, when using apply_async (#10676) The instrumentation for the Celery integration relies on various [Celery signals ](https://docs.celeryq.dev/en/stable/userguide/signals.html) in order to start and end the span when calling on `apply_async`. The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces). **Example:** - dd-trace-py expects the signal `before_task_publish` to start the span then `after_task_publish` to close the span. If the `after_task_publish` signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish. - The same thing above can also happen to `task_prerun` and `task_postrun`. **Solution** This PR patches `apply_async` so that there is a check to see if there is a span lingering around and closes it when `apply_task` is called. If an internal exception happens, the error will be marked on the `celery.apply` span. To track this, I added new logs in debug mode: > The after_task_publish signal was not called, so manually closing span and > The task_postrun signal was not called, so manually closing span There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not. Special Thanks: - Thanks to @tabgok for going through this with me in great detail! - @timmc-edx for helping us track it down! [APMS-13158] ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) APMS-13158 [APMS-13158]: https://datadoghq.atlassian.net/browse/APMS-13158?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Emmett Butler <[email protected]> (cherry picked from commit 0d28e08)

@tabgok

…sh, when using apply_async (#10676) The instrumentation for the Celery integration relies on various [Celery signals ](https://docs.celeryq.dev/en/stable/userguide/signals.html) in order to start and end the span when calling on `apply_async`. The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces). **Example:** - dd-trace-py expects the signal `before_task_publish` to start the span then `after_task_publish` to close the span. If the `after_task_publish` signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish. - The same thing above can also happen to `task_prerun` and `task_postrun`. **Solution** This PR patches `apply_async` so that there is a check to see if there is a span lingering around and closes it when `apply_task` is called. If an internal exception happens, the error will be marked on the `celery.apply` span. To track this, I added new logs in debug mode: > The after_task_publish signal was not called, so manually closing span and > The task_postrun signal was not called, so manually closing span There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not. Special Thanks: - Thanks to @tabgok for going through this with me in great detail! - @timmc-edx for helping us track it down! [APMS-13158] ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) APMS-13158 [APMS-13158]: https://datadoghq.atlassian.net/browse/APMS-13158?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Emmett Butler <[email protected]> (cherry picked from commit 0d28e08)

@tabgok

…sh, when using apply_async (#10676) The instrumentation for the Celery integration relies on various [Celery signals ](https://docs.celeryq.dev/en/stable/userguide/signals.html) in order to start and end the span when calling on `apply_async`. The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces). **Example:** - dd-trace-py expects the signal `before_task_publish` to start the span then `after_task_publish` to close the span. If the `after_task_publish` signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish. - The same thing above can also happen to `task_prerun` and `task_postrun`. **Solution** This PR patches `apply_async` so that there is a check to see if there is a span lingering around and closes it when `apply_task` is called. If an internal exception happens, the error will be marked on the `celery.apply` span. To track this, I added new logs in debug mode: > The after_task_publish signal was not called, so manually closing span and > The task_postrun signal was not called, so manually closing span There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not. Special Thanks: - Thanks to @tabgok for going through this with me in great detail! - @timmc-edx for helping us track it down! [APMS-13158] ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) APMS-13158 [APMS-13158]: https://datadoghq.atlassian.net/browse/APMS-13158?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ --------- Co-authored-by: Emmett Butler <[email protected]> (cherry picked from commit 0d28e08)

@timmc-edx

… 2.12] (#10872) Backport 6346fcb from #10848 to 2.12. This PR fixes an issue where Celery's closing signals got triggered but dd-trace-py skipped closing the `celery.apply` span due to not finding the task id. In celery's `task_protocol: 1`, the id is in the message of the body: https://docs.celeryq.dev/en/main/internals/protocol.html#message-body . The issue with the previous logic is that if the headers does have information (even if the headers were unrelated to the id), it would skip the check of the id in the body: before: ``` if headers: ``` after (this PR): ``` if headers and 'id' in headers: ``` By doing this, we check the headers for the id, then check the body for the id. If it fails to find the task id in the body or header, then it still hits the debug log, `unable to extract the Task and the task_id. This version of Celery may not be supported.` . This PR relates to the goal of #10676 , to close celery spans. If for some reason the logic in this PR fails to close an open `celery.apply` span, #10676 will act as a fail safe and close it. Special Thanks: @timmc-edx for helping us track this down! ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: wantsui <[email protected]>

@timmc-edx

… 2.13] (#10873) Backport 6346fcb from #10848 to 2.13. This PR fixes an issue where Celery's closing signals got triggered but dd-trace-py skipped closing the `celery.apply` span due to not finding the task id. In celery's `task_protocol: 1`, the id is in the message of the body: https://docs.celeryq.dev/en/main/internals/protocol.html#message-body . The issue with the previous logic is that if the headers does have information (even if the headers were unrelated to the id), it would skip the check of the id in the body: before: ``` if headers: ``` after (this PR): ``` if headers and 'id' in headers: ``` By doing this, we check the headers for the id, then check the body for the id. If it fails to find the task id in the body or header, then it still hits the debug log, `unable to extract the Task and the task_id. This version of Celery may not be supported.` . This PR relates to the goal of #10676 , to close celery spans. If for some reason the logic in this PR fails to close an open `celery.apply` span, #10676 will act as a fail safe and close it. Special Thanks: @timmc-edx for helping us track this down! ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: wantsui <[email protected]>

@tabgok

…sh, when using apply_async [backport 2.14] (#10893) Backport 0d28e08 from #10676 to 2.14. The instrumentation for the Celery integration relies on various [Celery signals ](https://docs.celeryq.dev/en/stable/userguide/signals.html) in order to start and end the span when calling on `apply_async`. The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces). **Example:** - dd-trace-py expects the signal `before_task_publish` to start the span then `after_task_publish` to close the span. If the `after_task_publish` signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish. - The same thing above can also happen to `task_prerun` and `task_postrun`. **Solution** This PR patches `apply_async` so that there is a check to see if there is a span lingering around and closes it when `apply_task` is called. If an internal exception happens, the error will be marked on the `celery.apply` span. To track this, I added new logs in debug mode: > The after_task_publish signal was not called, so manually closing span and > The task_postrun signal was not called, so manually closing span There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not. Special Thanks: - Thanks to @tabgok for going through this with me in great detail! - @timmc-edx for helping us track it down! [APMS-13158] ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) APMS-13158 [APMS-13158]: https://datadoghq.atlassian.net/browse/APMS-13158?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: wantsui <[email protected]>

@tabgok

…sh, when using apply_async [backport 2.13] (#10892) Backport 0d28e08 from #10676 to 2.13. The instrumentation for the Celery integration relies on various [Celery signals ](https://docs.celeryq.dev/en/stable/userguide/signals.html) in order to start and end the span when calling on `apply_async`. The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces). **Example:** - dd-trace-py expects the signal `before_task_publish` to start the span then `after_task_publish` to close the span. If the `after_task_publish` signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish. - The same thing above can also happen to `task_prerun` and `task_postrun`. **Solution** This PR patches `apply_async` so that there is a check to see if there is a span lingering around and closes it when `apply_task` is called. If an internal exception happens, the error will be marked on the `celery.apply` span. To track this, I added new logs in debug mode: > The after_task_publish signal was not called, so manually closing span and > The task_postrun signal was not called, so manually closing span There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not. Special Thanks: - Thanks to @tabgok for going through this with me in great detail! - @timmc-edx for helping us track it down! [APMS-13158] ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) APMS-13158 [APMS-13158]: https://datadoghq.atlassian.net/browse/APMS-13158?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: wantsui <[email protected]> Co-authored-by: erikayasuda <[email protected]>

@tabgok

…sh, when using apply_async [backport 2.12] (#10891) Backport 0d28e08 from #10676 to 2.12. The instrumentation for the Celery integration relies on various [Celery signals ](https://docs.celeryq.dev/en/stable/userguide/signals.html) in order to start and end the span when calling on `apply_async`. The integration can fail if the expected signals don't trigger, which can lead to broken context propagation (and unexpected traces). **Example:** - dd-trace-py expects the signal `before_task_publish` to start the span then `after_task_publish` to close the span. If the `after_task_publish` signal never gets called (which can happen if a Celery exception occurs while processing the app), then the span won't finish. - The same thing above can also happen to `task_prerun` and `task_postrun`. **Solution** This PR patches `apply_async` so that there is a check to see if there is a span lingering around and closes it when `apply_task` is called. If an internal exception happens, the error will be marked on the `celery.apply` span. To track this, I added new logs in debug mode: > The after_task_publish signal was not called, so manually closing span and > The task_postrun signal was not called, so manually closing span There's a related PR #10848 that works to improve how we extract information based on the protocols, that also affects when spans get closed or not. Special Thanks: - Thanks to @tabgok for going through this with me in great detail! - @timmc-edx for helping us track it down! [APMS-13158] ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) APMS-13158 [APMS-13158]: https://datadoghq.atlassian.net/browse/APMS-13158?atlOrigin=eyJpIjoiNWRkNTljNzYxNjVmNDY3MDlhMDU5Y2ZhYzA5YTRkZjUiLCJwIjoiZ2l0aHViLWNvbS1KU1cifQ Co-authored-by: wantsui <[email protected]> Co-authored-by: erikayasuda <[email protected]>

iherasymenko · 2024-12-05T01:39:21Z

Linking to #11624 as this PR introduced a regression.

…chains scenario (#11498) We've made a few changes to handle celery context recently, including: #10676 In particular the goal of #10676 was to handle a scenario where a long running task may run into an exception, preventing it from closing. Unfortunately, this scenario did not account for cases where tasks are chained and may not close until later. See: #11479 and #11624 With this PR, the sample app in #11479 would attach the celery specific span back to the root span. I also need to add tests for the chains scenario. Related to AIDM-494 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

…chains scenario (#11498) We've made a few changes to handle celery context recently, including: #10676 In particular the goal of #10676 was to handle a scenario where a long running task may run into an exception, preventing it from closing. Unfortunately, this scenario did not account for cases where tasks are chained and may not close until later. See: #11479 and #11624 With this PR, the sample app in #11479 would attach the celery specific span back to the root span. I also need to add tests for the chains scenario. Related to AIDM-494 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) (cherry picked from commit e8aab65)

…chains scenario [backport 2.18] (#11806) Backport e8aab65 from #11498 to 2.18. We've made a few changes to handle celery context recently, including: #10676 In particular the goal of #10676 was to handle a scenario where a long running task may run into an exception, preventing it from closing. Unfortunately, this scenario did not account for cases where tasks are chained and may not close until later. See: #11479 and #11624 With this PR, the sample app in #11479 would attach the celery specific span back to the root span. I also need to add tests for the chains scenario. Related to AIDM-494 ## Checklist - [x] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [x] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) Co-authored-by: wantsui <[email protected]>

fix(celery) close apply_async spans even when the after_task_publish_…

7baa331

…signal does not get called

wantsui requested a review from tabgok September 16, 2024 21:13

Update format.

3d21563

wantsui changed the title ~~fix(celery) Close celery.apply spans when using apply_async, even when after_task_publish isn't called~~ fix(celery): Close celery.apply spans even without after_task_publish, when using apply_async Sep 17, 2024

wantsui changed the title ~~fix(celery): Close celery.apply spans even without after_task_publish, when using apply_async~~ fix(celery): close celery.apply spans even without after_task_publish, when using apply_async Sep 17, 2024

wantsui added 4 commits September 17, 2024 14:20

Merge branch 'main' into close-apply-async-celery-spans

b067390

Add celery.app.task.Task to the Celery Base Test Case

64abe8f

Remove pin from celery.app.task.Task and update patch and tests accor…

f0c2892

…dingly.

Comment out new closing span changes and add a test showing that if t…

e5d124f

…he after task publish signal is not called, there are no spans at the moment so the test fails.

datadog-datadog-prod-us1 bot reviewed Sep 19, 2024

View reviewed changes

tests/contrib/celery/test_integration.py Outdated Show resolved Hide resolved

wantsui added 11 commits September 19, 2024 17:13

Add new change to close celery.apply spans back, with debug log messages

a9a234e

Simplify conditions in utils and test.

c0d387d

Merge branch 'main' into close-apply-async-celery-spans

f4400ee

update with latest changes on the main branch

Remove unused sys import.

18af2fa

Fix typo.

5783790

Apply formatting suggestions.

16e4424

Fix more formatting.

444b074

Add release note describing the fix to celery.apply.

40293ec

Add additional assertions for span metadata.

7f54148

Merge branch 'main' into close-apply-async-celery-spans

0606be9

add changes from main branch

Fix typo.

2c2db93

wantsui marked this pull request as ready for review September 20, 2024 15:50

wantsui requested review from a team as code owners September 20, 2024 15:50

wantsui requested review from P403n1x87 and emmettbutler September 20, 2024 15:50

wantsui added the backport 2.14 label Sep 30, 2024

tabgok approved these changes Oct 1, 2024

View reviewed changes

wantsui enabled auto-merge (squash) October 1, 2024 17:50

timmc-edx mentioned this pull request Oct 1, 2024

Disabling NR APM causes trace concatenation in Datadog edx/edx-arch-experiments#692

Closed

22 tasks

Merge branch 'main' into close-apply-async-celery-spans

81b2eb9

wantsui merged commit 0d28e08 into main Oct 1, 2024
576 checks passed

wantsui deleted the close-apply-async-celery-spans branch October 1, 2024 20:04

github-actions bot mentioned this pull request Oct 1, 2024

fix(celery): close celery.apply spans even without after_task_publish, when using apply_async [backport 2.12] #10891

Merged

2 tasks

github-actions bot mentioned this pull request Oct 1, 2024

fix(celery): close celery.apply spans even without after_task_publish, when using apply_async [backport 2.13] #10892

Merged

2 tasks

github-actions bot mentioned this pull request Oct 1, 2024

fix(celery): close celery.apply spans even without after_task_publish, when using apply_async [backport 2.14] #10893

Merged

2 tasks

oranav mentioned this pull request Nov 21, 2024

Celery task_postrun is called only after the span has finished in chains (due to a regression) #11479

Open

wantsui mentioned this pull request Nov 21, 2024

fix(celery): stop closing prerun_span too soon to account for Celery chains scenario #11498

Merged

2 tasks

github-actions bot mentioned this pull request Dec 19, 2024

fix(celery): stop closing prerun_span too soon to account for Celery chains scenario [backport 2.16] #11804

Open

2 tasks

github-actions bot mentioned this pull request Dec 19, 2024

fix(celery): stop closing prerun_span too soon to account for Celery chains scenario [backport 2.17] #11805

Open

2 tasks

github-actions bot mentioned this pull request Dec 19, 2024

fix(celery): stop closing prerun_span too soon to account for Celery chains scenario [backport 2.18] #11806

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(celery): close `celery.apply` spans even without after_task_publish, when using apply_async #10676

fix(celery): close `celery.apply` spans even without after_task_publish, when using apply_async #10676

wantsui commented Sep 16, 2024 •

edited by tabgok

Loading

datadog-dd-trace-py-rkomorn bot commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

pr-commenter bot commented Sep 16, 2024 •

edited

Loading

iherasymenko commented Dec 5, 2024

fix(celery): close celery.apply spans even without after_task_publish, when using apply_async #10676

fix(celery): close celery.apply spans even without after_task_publish, when using apply_async #10676

Conversation

wantsui commented Sep 16, 2024 • edited by tabgok Loading

Checklist

Reviewer Checklist

datadog-dd-trace-py-rkomorn bot commented Sep 16, 2024 • edited Loading

Datadog Report

github-actions bot commented Sep 16, 2024 • edited Loading

pr-commenter bot commented Sep 16, 2024 • edited Loading

Benchmarks

iherasymenko commented Dec 5, 2024

fix(celery): close `celery.apply` spans even without after_task_publish, when using apply_async #10676

fix(celery): close `celery.apply` spans even without after_task_publish, when using apply_async #10676

wantsui commented Sep 16, 2024 •

edited by tabgok

Loading

datadog-dd-trace-py-rkomorn bot commented Sep 16, 2024 •

edited

Loading

github-actions bot commented Sep 16, 2024 •

edited

Loading

pr-commenter bot commented Sep 16, 2024 •

edited

Loading