Deadlock when using ProcessPoolExecutor with stack v2 profiler

[oranav]

We've encountered deadlocks which prevent the process from exiting successfully when using the V2 stack profiler.
Python version is 3.11.11, ddtrace version is 2.17.3, architecture is linux/amd64.

The child process is stuck on Datadog::Sampler::register_thread:

(gdb) bt
#0  futex_wait (private=0, expected=2, futex_word=0x7f78af07e760) at ../sysdeps/nptl/futex-internal.h:146
#1  __GI___lll_lock_wait (futex=futex@entry=0x7f78af07e760, private=0) at ./nptl/lowlevellock.c:49
#2  0x00007f78b0f2b452 in lll_mutex_lock_optimized (mutex=0x7f78af07e760) at ./nptl/pthread_mutex_lock.c:48
#3  ___pthread_mutex_lock (mutex=0x7f78af07e760) at ./nptl/pthread_mutex_lock.c:93
#4  0x00007f78af06ea8a in ?? ()
   from /app/venv/lib/python3.11/site-packages/ddtrace/internal/datadog/profiling/stack_v2/_stack_v2.cpython-311-x86_64-linux-gnu.so
#5  0x00007f78af074269 in Datadog::Sampler::register_thread(unsigned long, unsigned long, char const*) ()
   from /app/venv/lib/python3.11/site-packages/ddtrace/internal/datadog/profiling/stack_v2/_stack_v2.cpython-311-x86_64-linux-gnu.so

And then the parent process waits in join() forever.

Here's a simple reproduction script:

import time
from concurrent.futures import ProcessPoolExecutor


def worker():
    time.sleep(0.1)


def main():
    with ProcessPoolExecutor(max_workers=16) as executor:
        for _ in range(128):
            executor.submit(worker)


if __name__ == "__main__":
    main()

Run with:

DD_PROFILING_ENABLED=1 DD_PROFILING_STACK_V2_ENABLED=1 ddtrace-run python3 test.py

Just run a couple of times (it's non-deterministic), and you'll see hangups every now and then.

I've also noticed that the stuck process is always the first worker process created - might lead to some clue.

Thanks!

[sanchda]

Whoa! Very meaningful issue, @oranav. Deepest apologies for the inconvenience, but thank you for the report. Will work on this ASAP.

[sanchda]

Took a careful look at this, and thanks to some insight from @taegyunkim I think #11768 fixes it.

[sanchda]

Will update this thread with

1. Timeline for release. Note that the release will be backported to 2.16 onward.
2. A potential strategy for pulling a pre-release artifact for deployment in affected environments (not sure how durable this will be--but hey, if your application is deadlocking, then it's either disable the feature, wait for a release, or do something else, right?)

@oranav; thanks again for the comprehensive overview, the thoughtful discussion, and the effective reproduction. This really improves our ability to provide a speedy fix. 🙇

[oranav]

Thanks @sanchda! Indeed one of my suspicions was that the process forks while the mutex is held, but I thought forking must happen while the GIL is taken, and I thought Echion's mutices are all taken under GIL - but apparently I missed something :)

[sanchda]

#11768 has been merged, so this should be fixed in an upcoming release.

but I thought forking must happen while the GIL is taken, and I thought Echion's mutices are all taken under GIL
You're correct that os.fork() will happen under the GIL, but as you suspect there are a few wrinkles.

• Our use of Echion places it on a native thread, where it is not beholden to the GIL in any way. In particular, that means the state of each mutex is indeterminate at time of fork() and we have to do... uh... stuff. To make the world right again.

• That pattern must extend to all of the other native mutexes in the project.

[sanchda]

I'm going to keep this open until an end-user reports that the new release fixes the problem. @oranav ; if you like, I can direct you to a workflow for getting access to a pre-release build, but otherwise I can update this thread when a known official release has been made available.

[sanchda]

FYI, this will be released (later today, I hope) in 2.18.1. I'm also attempting to back-port to the 2.17 and 2.16 lines (🤞). It'll be part of mainline starting in the 2.19.0 release.

[sanchda]

Confirming that 2.18.1 shipped. Would love to hear some folks weigh in on whether or not it solved this problem for them.