Daemon threads are not forced to exit as part of runtime finalization

[mpage]

Bug report

Bug description:

Daemon threads are forced to exit during finalization in default builds:

cpython/Python/pylifecycle.c

Lines 2023 to 2026 in 10de360

	/* Remaining daemon threads will automatically exit
	when they attempt to take the GIL (ex: PyEval_RestoreThread()). */
	_PyInterpreterState_SetFinalizing(tstate->interp, tstate);
	_PyRuntimeState_SetFinalizing(runtime, tstate);

Such threads will exit immediately the next time that they acquire the GIL:

cpython/Python/ceval_gil.c

Lines 294 to 302 in 10de360

	if (_PyThreadState_MustExit(tstate)) {
	/* bpo-39877: If Py_Finalize() has been called and tstate is not the
	thread which called Py_Finalize(), exit immediately the thread.
	This code path can be reached by a daemon thread after Py_Finalize()
	completes. In this case, tstate is a dangling pointer: points to
	PyThreadState freed memory. */
	PyThread_exit_thread();
	}

I don't believe daemon threads are forced to exit in the free-threaded builds. If you run the following script under GDB:

import threading


def loop():
    while True:
        pass


thr = threading.Thread(target=loop, daemon=True)
thr.start()

and break immediately before _Py_Finalize returns, the daemon thread will be blocked in tstate_wait_attach:

(gdb) fin
Run till exit from #0  _PyRuntime_Finalize () at Python/pylifecycle.c:133
_Py_Finalize (runtime=runtime@entry=0xa3d4c0 <_PyRuntime>) at Python/pylifecycle.c:2204
2204	    return status;
(gdb) p status
$1 = 0
(gdb) info threads
  Id   Target Id            Frame
* 1    LWP 1330531 "python" _Py_Finalize (runtime=runtime@entry=0xa3d4c0 <_PyRuntime>) at Python/pylifecycle.c:2204
  2    LWP 1330532 "python" 0x00007ffff7c8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6
(gdb) thread 2
[Switching to thread 2 (LWP 1330532)]
#0  0x00007ffff7c8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff7c8679a in __futex_abstimed_wait_common () from /lib64/libc.so.6
#1  0x00007ffff7c91c48 in __new_sem_wait_slow64.constprop.0 () from /lib64/libc.so.6
#2  0x00000000006ec06f in _PySemaphore_PlatformWait (sema=sema@entry=0x7ffff79ff8b0, timeout=timeout@entry=-1) at Python/parking_lot.c:142
#3  0x00000000006ec188 in _PySemaphore_Wait (sema=sema@entry=0x7ffff79ff8b0, timeout=timeout@entry=-1, detach=detach@entry=0) at Python/parking_lot.c:213
#4  0x00000000006ec2e5 in _PyParkingLot_Park (addr=addr@entry=0xb29618, expected=expected@entry=0x7ffff79ff93c, size=size@entry=4, timeout_ns=timeout_ns@entry=-1, park_arg=park_arg@entry=0x0, detach=detach@entry=0)
    at Python/parking_lot.c:316
#5  0x00000000006fd6ee in tstate_wait_attach (tstate=tstate@entry=0xb295f0) at Python/pystate.c:2096
#6  0x00000000006ff645 in _PyThreadState_Attach (tstate=tstate@entry=0xb295f0) at Python/pystate.c:2126
#7  0x00000000006bde6b in _Py_HandlePending (tstate=0xb295f0) at Python/ceval_gil.c:1261
#8  0x000000000066fe3e in _PyEval_EvalFrameDefault (tstate=tstate@entry=0xb295f0, frame=0x7ffff7e7b1a8, throwflag=throwflag@entry=0) at Python/generated_cases.c.h:4720
#9  0x000000000067d992 in _PyEval_EvalFrame (throwflag=0, frame=<optimized out>, tstate=0xb295f0) at ./Include/internal/pycore_ceval.h:119
#10 _PyEval_Vector (tstate=<optimized out>, func=<optimized out>, locals=locals@entry=0x0, args=0x7ffff79ffd08, argcount=1, kwnames=0x0) at Python/ceval.c:1848
#11 0x00000000004c4ab2 in _PyFunction_Vectorcall (func=<optimized out>, stack=<optimized out>, nargsf=<optimized out>, kwnames=<optimized out>) at Objects/call.c:413
#12 0x00000000004c9211 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=1, args=0x7ffff79ffd08, callable=<function at remote 0x20000aa5cd0>, tstate=0xb295f0) at ./Include/internal/pycore_call.h:167
#13 method_vectorcall (method=<optimized out>, args=<optimized out>, nargsf=<optimized out>, kwnames=0x0) at Objects/classobject.c:70
#14 0x00000000004c726d in _PyVectorcall_Call (tstate=tstate@entry=0xb295f0, func=0x4c8f7f <method_vectorcall>, callable=callable@entry=<method at remote 0x2000034a4b0>, tuple=tuple@entry=(), kwargs=kwargs@entry=0x0)
    at ./Include/object.h:763
#15 0x00000000004c75d9 in _PyObject_Call (tstate=0xb295f0, callable=<method at remote 0x2000034a4b0>, args=(), kwargs=0x0) at Objects/call.c:348
#16 0x00000000004c7630 in PyObject_Call (callable=<optimized out>, args=<optimized out>, kwargs=<optimized out>) at Objects/call.c:373
#17 0x00000000007ab620 in thread_run (boot_raw=boot_raw@entry=0xb27f10) at ./Modules/_threadmodule.c:345
#18 0x000000000071c7e0 in pythread_wrapper (arg=<optimized out>) at Python/thread_pthread.h:243
#19 0x00007ffff7c89c02 in start_thread () from /lib64/libc.so.6
#20 0x00007ffff7d0ec40 in clone3 () from /lib64/libc.so.6
(gdb)

Assuming this is correct, finalizers that run during runtime finalization cannot safely join daemon threads. I'm not sure if this is something we want to support in the free-threaded build, but I thought it was worth documenting since it's a behavioral difference from the default build.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux