after using funlib.persistence 0.5.3 multiprocess lambda error

[mzouink]

After start using the new funlib.persistence. in a new envirement i got this bug where training.

AttributeError: Can't pickle local object 'IntensitiesArrayConfig.array.<locals>.<lambda>'
  File "/home/[email protected]/miniconda3/envs/dacapo/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
Traceback (most recent call last):
  File "/home/[email protected]/miniconda3/envs/dacapo/lib/python3.11/multiprocessing/queues.py", line 244, in _feed
    obj = _ForkingPickler.dumps(obj)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/[email protected]/miniconda3/envs/dacapo/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'IntensitiesArrayConfig.array.<locals>.<lambda>'

Got the bug with and without multiprocessing.set_start_method("fork", force=True)

python 3.11
OS: Ubuntu 22.04.4 LTS
@pattonw

[mzouink]

after testing:

Error happens only with distance task:

same tutorial only difference is using DistanceTask:
example: https://gist.github.com/mzouink/0fc5a0f9627591833a62df68c92c681e

[mzouink]

very weird behavior:
when i install using pypi : pip install dacapo-ml it doesn't work: https://pypi.org/project/dacapo-ml/
when i install with pip install git+https://github.com/janelia-cellmap/dacapo.git it doesn't work
but when i clone and i install pip install -e . it works

[pattonw]

Hmm, Ok this is strange behavior. I can take a look. I am very surprised that it doesn't work even with setting multiprocessing spawn method to "fork" since there shouldn't be a need to pickle the local lambda function at all if using the "fork" spawn method

[pattonw]

I don't get any pickling errors running that tutorial anymore. I suspect the issue was caused by some other bug that was fixed when we expanded the tests, and only presented as a pickling error when the the worker tried to pass the exception back to the watchdog process.

I tried installing from github main branch, and installing locally, both worked fine for me.

[mzouink]

as you said last time it can be a bug in the code but get reported as lambda serialization problem.

I will update everything and if i don't get the error anymore i will close the issue