Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[hma] ValueError in indexer #1345

Open
Dcallies opened this issue Aug 16, 2023 · 0 comments
Open

[hma] ValueError in indexer #1345

Dcallies opened this issue Aug 16, 2023 · 0 comments
Labels
bug python-threatexchange Items related to the threatexchange python tool / library

Comments

@Dcallies
Copy link
Contributor

Dcallies commented Aug 16, 2023

Indexer fails to build

[ERROR] ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (264538,) + inhomogeneous part.
Traceback (most recent call last):
  File "/var/task/hmalib/lambdas/unified_indexer.py", line 132, in lambda_handler
    index: SignalTypeIndex = signal_type.get_index_cls().build(merged_data)
  File "/var/lang/lib/python3.8/site-packages/threatexchange/signal_type/index.py", line 216, in build
    ret.add_all(entries)
  File "/var/lang/lib/python3.8/site-packages/threatexchange/signal_type/pdq/pdq_index.py", line 75, in add_all
    self.index.add(
  File "/var/lang/lib/python3.8/site-packages/threatexchange/signal_type/pdq/pdq_faiss_matcher.py", line 240, in add
    self.faiss_index.add_with_ids(numpy.array(vectors), numpy.array(i64_ids))

Likely there's a malformed PDQ hash in the inputs somehow. PDQ index was written before we figured out how to correctly use the faiss interface, so there might be some other issue.

@Dcallies Dcallies added bug hma Items related to the hasher-matcher-actioner system labels Aug 16, 2023
@Dcallies Dcallies added python-threatexchange Items related to the threatexchange python tool / library and removed hma Items related to the hasher-matcher-actioner system labels Mar 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug python-threatexchange Items related to the threatexchange python tool / library
Projects
None yet
Development

No branches or pull requests

1 participant