Bugfix: multiprocessing queue doesn't pickle _size #107
Conversation
Solved by adding __getstate__ and __setstate__ methods
util/queue.py
Outdated
| @@ -66,7 +66,7 @@ class Queue(multiprocessing.queues.Queue): | |||
| """ | |||
|
|
|||
| def __init__(self, *args, **kwargs): | |||
| super(Queue, self).__init__(*args, **kwargs) | |||
| super(Queue, self).__init__(*args, **kwargs, ctx=multiprocessing.get_context()) | |||
There was a problem hiding this comment.
Please list this before **kwargs, so that it also work with Python 2 (see the failing Travis CI test).
|
Thanks for contributing this! Please see my comments. |
- fix legacy python compatibility - use namedtuple to pass state information
|
Sorry about the delay! I've made the requested changes. (Edit: and fixed the missing import. Sorry about that.) |
| @@ -66,9 +71,20 @@ class Queue(multiprocessing.queues.Queue): | |||
| """ | |||
|
|
|||
| def __init__(self, *args, **kwargs): | |||
| if sys.version_info >= (3, 4) and 'ctx' not in kwargs: | |||
There was a problem hiding this comment.
Please add a comment here explaining why this is necessary (i.e. why we need multiprocessing.get_context() in Python 3.4+).
| @@ -51,6 +53,9 @@ def value(self): | |||
| return self.count.value | |||
|
|
|||
|
|
|||
| QueueState = namedtuple('QueueState', ['queue', 'size']) | |||
|
|
|||
There was a problem hiding this comment.
Let's make this class non-public: _QueueState.
| super(Queue, self).__init__(*args, **kwargs) | ||
| self._size = SharedCounter(0) | ||
|
|
||
| # __getstate__ and __setstate__ are needed for pickling, otherwise _size won't be copied. |
There was a problem hiding this comment.
Let's drop this comment and instead add docstrings to the two methods, e.g. "Returns the contents to pickle for the instance" and "Sets the state of the instance upon unpickling".
I needed to make these small changes in order to get queue working with my multiprocessing code in python 3.6+.
There were two issues:
multiprocessing.queues.Queue.__init__takes actxargument, which was not provided_sizedoesn't get pickled, and get/put/etc will throw an AttributeError about there being no such attribute