Multiprocessing Pool Crash

[jadczak]

Attempting to profile programs with multiprocessing yields the following error:

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed

I was able to reproduce the error using the example from #809 (comment)

import multiprocessing as mp

def fun(args):
    x = 0
    for i in range(100):
        x += 1
    return args

if __name__ == "__main__":
    with mp.Pool(mp.cpu_count()) as p:
        r = p.map(fun, range(1000000))
    print(sum(x for x in r))
    print(((1000000-1) * 1000000) / 2)

Expected behavior
No crash?

Screenshots

Desktop (please complete the following information):

• OS: Windows 11
• Browser Firefox
• Version 129
• Python: CPython 3.12.3
• Scalene: Repository version

Error Text

Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "C:\Users\||||||||\scratch\py312\Lib\site-packages\scalene\scalene_profiler.py", line 1791, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\||||||||\scratch\scalene_mp.py", line 10, in <module>
    with mp.Pool(mp.cpu_count()) as p:
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 215, in __init__
    self._repopulate_pool()
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 306, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 329, in _repopulate_pool_static
    w.start()
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\context.py", line 337, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\popen_spawn_win32.py", line 95, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Scalene: The specified code did not run for long enough to profile.
By default, Scalene only profiles code in the file executed and its subdirectories.
To track the time spent in all files, use the `--profile-all` option.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\spawn.py", line 113, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\||||||||\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
           ^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 6] The handle is invalid

[emeryberger]

• Works fine on Mac OS X, repository version of Scalene.
• Found an occasional race condition on exit on Ubuntu 20, but not the issue mentioned above (perhaps it's a WSL thing)

[jadczak]

This was on my windows machine not running in WSL. My wild shot in the dark is maybe a fork vs spawn thing since windows doesn't support fork 🤷

Once I get back to my windows machine I'll give WSL a try and see if I run into similar problems. It doesn't look like python 3.12.4 fixed anything relevant to multiprocessing, but I'll give a few other versions a try to see if there is something funky going on with my particular version of python.

[jadczak]

Tested the following python versions on Windows:
3.12.5 - OSError every time
3.11.9 - OSError every time

Tested the following python version on Clear Linux
3.12.4 - Worked every time tested

Test the following python version WSL
3.11.2 - Frequently hangs. No error when it finishes.

[hnguyenHWI]

I'm seeing something similar. Windows 11 22H2, Python 3.10.2, Scalene version 1.5.45 (2024.10.01)

I was also able to reproduce from #809 (comment).

main.py (stripped down code from the app for a client)

import multiprocessing
import sys
import time
import traceback


class AppLog:
    """
    Implements configuration and handling of logging for XXXXXX application. Provides
    the target to be invoked in a separate process."""

    def __init__(
        self,
        log_queue: multiprocessing.Queue,
    ) -> None:
        self.log_queue = log_queue
        # Configure loggers, filters, handlers, then watches the queue and writes the
        # queued messages to different files
        self.__loop_handle_log_queue()

    def __loop_handle_log_queue(self) -> None:
        while True:
            try:
                # Loop forever, block until a message is put onto the queue by a
                # QueueHandler from a different process, handle the message and write
                # it out to the appropriate file
                time.sleep(1)
                continue
            except Exception:
                print("Logging exception: ", file=sys.stderr)
                traceback.print_exc(file=sys.stderr)


def __log_proc(
    log_queue: multiprocessing.Queue,
):
    AppLog(log_queue=log_queue)


def start_app_logging_process(
    log_queue: multiprocessing.Queue,
) -> multiprocessing.Process:
    """Target for logging process which will log to application-wide file."""

    process = multiprocessing.Process(
        target=__log_proc,
        args=(log_queue,),
    )
    process.start()
    return process


def main_wrapper():
    """Wraps the call to main so that we can use the Click library \
        (e.g. for option flags)."""

    # Now launching the app, show splash screen until launcher is initialized

    log_queue = multiprocessing.Queue(-1)
    log_process = start_app_logging_process(log_queue=log_queue)

    # Have started logging queue, move on to the rest of the application:
    # open more processes, do other stuff


def func_issue_809():
    def fun(args):
        x = 0
        for i in range(100):
            x += 1
        return args
    with multiprocessing.Pool(multiprocessing.cpu_count()) as p:
        r = p.map(fun, range(1000000))
    print(sum(x for x in r))
    print(((1000000-1) * 1000000) / 2)

if __name__ == "__main__":
    # disable pylint, need this wrapper so Click will work
    func_issue_809()
    main_wrapper()  # pylint:disable=no-value-for-parameter

Error output from the code from 809

(venv) PS C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT> scalene --version
Scalene version 1.5.45 (2024.10.01)
(venv) PS C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT> scalene .\scalene_minimal\main.py
Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\venv\lib\site-packages\scalene\scalene_profiler.py", line 1795, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 79, in <module>
    func_issue_809()
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 72, in func_issue_809
    with multiprocessing.Pool(multiprocessing.cpu_count()) as p:
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 119, in Pool
    return Pool(processes, initializer, initargs, maxtasksperchild,
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 212, in __init__
    self._repopulate_pool()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 303, in _repopulate_pool
    return self._repopulate_pool_static(self._ctx, self.Process,
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\pool.py", line 326, in _repopulate_pool_static
    w.start()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid

Error output from my minimal code, main_wrapper() function in main.py above

(venv) PS C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT> scalene .\scalene_minimal\main.py
Error in program being profiled:
 Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\venv\lib\site-packages\scalene\scalene_profiler.py", line 1795, in profile_code
    exec(code, the_globals, the_locals)
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 67, in <module>
    main_wrapper()  # pylint:disable=no-value-for-parameter
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 60, in main_wrapper
    log_process = start_app_logging_process(log_queue=log_queue)
  File "C:\Users\USER\ORGANIZATION\customer\CLIENT1\CLIENT_PROJECT\scalene\PROJECT_ROOT\scalene_minimal\main.py", line 49, in start_app_logging_process
    process.start()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 224, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
_pickle.PicklingError: Can't pickle <class 'scalene.replacement_sem_lock.replacement_semlock.ReplacementSemLock'>: attribute lookup replacement_semlock.ReplacementSemLock on scalene.replacement_sem_lock failed
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\spawn.py", line 107, in spawn_main
    new_handle = reduction.duplicate(pipe_handle,
  File "C:\Users\USER\AppData\Local\Programs\Python\Python310\Lib\multiprocessing\reduction.py", line 79, in duplicate
    return _winapi.DuplicateHandle(
OSError: [WinError 6] The handle is invalid