Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for buffers in load_bytes() factory #11

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

lexplua
Copy link

@lexplua lexplua commented Sep 1, 2024

This PR addresses Bloom load_bytes support for all objects with buffer protocol implemented.
The problem I was trying to solve was:

  • Bloom filter creation in the main process
  • Passing existing filter to worker processes via Python SharedMemory
    The SharedMemory instance has .buf member, which is memoryview instance. Current Bloom implementation can work only with bytes to load serialized filter object. It leads to additional memory copy (in addition to one inside load method.

After this PR Bloom filter can be constructed from any object with buffer protocol support, such as memoryview, array, bytearray, numpy array (I guess)

On a side note it bumps Python ABI compatibility requirements to 3.11. From Pyo3 documentation about PyBuffer:

Available on non-Py_LIMITED_API or Py_3_11 only

@lexplua lexplua changed the title Support for buffers Support for buffers in load_bytes() factory Sep 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant