Handling of Celery root spans is broken

[iherasymenko]

Given:

pyproject.toml

[tool.poetry]
package-mode = false

[tool.poetry.dependencies]
python = "^3.12"
celery = { version = "5.4.0" }
#ddtrace = { git = "https://github.com/DataDog/dd-trace-py", rev = "3a859123249b5ecb00540a96243df2465bceb7c2" } # the commit that introduced the regression
#ddtrace = { git = "https://github.com/DataDog/dd-trace-py", rev = "758fb7dabcdd4db3e8aed7db6a74d639c217b1b1" } # last known working commit
ddtrace = "2.17.2" # The latest to date version; still broken
#ddtrace = "2.12.3" # The version the regression was introduced
#ddtrace = "2.12.2" # The last known working version

main.py

from celery import Celery
from ddtrace import tracer

app = Celery(
    'tasks',
    broker='amqp://localhost',
    broker_connection_retry_on_startup=True,
)

app.conf.accept_content = ['application/json', 'application/x-python-serialize']


@app.task(name='Task1')
def task1():
    print("Task 1 finished")


@app.task(name='Task2')
def task2():
    span1 = tracer.current_root_span()
    if span1 is not None:
        print("Writing my.tag to root span")
        span1.set_tag("my.tag", "my_value")
    else:
        print("Writing failed: No root span found")
    task1.delay()

    span2 = tracer.current_root_span()
    if span2 is not None:
        print(f"Reading my.tag: The value is {span2.get_tags().get('my.tag')}")
    else:
        print("Reading failed: No current span found")

    print("Task 2 finished")


if __name__ == '__main__':
    task2.delay()

Run:

poetry lock && poetry install --sync

and run:

poetry run ddtrace-run celery -A main worker

and then run:

poetry run ddtrace-run python main.py 

Version 2.12.2 and older:

[2024-12-04 19:05:17,051: WARNING/ForkPoolWorker-6] Writing my.tag to root span
[2024-12-04 19:05:17,060: WARNING/ForkPoolWorker-6] Reading my.tag: The value is my_value
[2024-12-04 19:05:17,060: WARNING/ForkPoolWorker-6] Task 2 finished
[2024-12-04 19:05:17,061: WARNING/ForkPoolWorker-7] Task 1 finished

Version 2.12.3 and newer:

[2024-12-04 19:12:27,720: WARNING/ForkPoolWorker-6] Writing my.tag to root span
[2024-12-04 19:12:27,738: WARNING/ForkPoolWorker-6] Reading failed: No current span found
[2024-12-04 19:12:27,738: WARNING/ForkPoolWorker-6] Task 2 finished
[2024-12-04 19:12:27,741: WARNING/ForkPoolWorker-7] Task 1 finished

[oranav]

I think this is essentially the same issue as #11479. In the other issue we trigger another task using a chain, here we just do it manually -- but it leads to the same consequence.

[wantsui]

@iherasymenko - Sorry for the change in behavior! Thanks for the reproduction app, and for flagging this with @oranav ! I just wrote this comment in #11479, but I'm taking a look at your example too to make sure we can capture this correctly in our tests without accidentally keeping the spans open indefinitely if an error occurs.

For now, I also recommend pinning your application to a different tracer, and I'll report back to both issues once I have an update.

[iherasymenko]

@wantsui thank you for looking into this. Unfortunately pinning the last known working version is not an option for us as it does not include a fix for #10714 .

[oranav]

@iherasymenko You can try my workaround here.

[wantsui]

As an update, I got the approvals I needed to get this merged. I'm also working to backport this fix to 2.18.0, 2.17.0, and 2.16.0. Would any of those versions work for you @iherasymenko ?

I know there was mention of targeting 2.14.0 for a pymongo issue, so just want to check in.

[iherasymenko]

I think 2.17 should work. Passing the torch to @Nanared1.

[wantsui]

As an update, we got 2.18.1 out with the celery update: v2.18.1 (release) . Can you give that a try?

Unfortunately, we can't release patches for 2.17.x or 2.16.x due to the holidays and code freezes, so that'll be out in January instead. I'll update this thread once we have those other versions out in January. Apologies for the inconvenience!