Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deprecate tokenize parameter in field definition #602

Merged
merged 2 commits into from
Oct 25, 2023

Conversation

aprudhomme
Copy link
Contributor

These are some proposed changes for the 1.0.0-SNAPSHOT branch around the handling of tokenization of text based fields.

  • The tokenize parameter in the field definition is deprecated and no longer used
  • TEXT type fields will always be tokenized
  • ATOM type fields will never be tokenized
  • ATOM norms are alway omitted, this is the default in ES. Ideally, we would want to make this configurable, but the grpc type does not allow for detecting when this is unset (and TEXT fields need the opposite default). Since this is not a commonly used option, I think hard coding the value is ok for now.
  • ATOM fields are no longer indexed for search when the search property is false

Additionally

  • references to the tokenize parameter have been removed
  • the parameter has been removed from schema files
  • the search parameter has been added to ATOM fields storing doc_id values in schemas, as they are no longer searchable without it. It would be good to switch these to _ID fields at some point, but that is beyond the scope of this branch.

@aprudhomme aprudhomme requested a review from sarthakn7 October 19, 2023 20:08
@aprudhomme
Copy link
Contributor Author

aprudhomme commented Oct 20, 2023

The example plugin test is an expected failure, since it builds against the latest published version

@aprudhomme aprudhomme merged commit 3027a0c into Yelp:1.0.0-SNAPSHOT Oct 25, 2023
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants