[Feature] Pagination in hybrid query #963

vibrantvarun · 2024-10-23T17:19:51Z

Description

This PR contains changes for enabling support for pagination in hybrid query.
The highlight of this PR are

Introduction of a new parameter "pagination_depth" to set a reference of hybrid query search results on which pagination can be applied.
Handling of single shard scenario where fetch phase can run before the normalization process.
Handling of from parameter conditions in Normalization processor.
Disabling scroll operation in hybrid query.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

New functionality includes testing.
New functionality has been documented.
API changes companion pull request created.
Commits are signed per the DCO using --signoff.
Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Varun Jain <[email protected]>

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

martin-gaievski · 2024-10-24T04:39:59Z

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

+        int paginationDepth;
+        HybridQuery hybridQuery;
+        Query query = searchContext.query();
+        if (query instanceof BooleanQuery) {


when such scenario can happen? we should not allow hybrid if it's not the top clause

I have added the comment on top why it is written like that. Basically in case of nested fields and alias filter, hybrid query gets wrapped under bool query.

simple case with nested fields is taken care of in QueryPhaseSearcher class. This logic doesn't belong here, unless this is a special scenario that we missed in that extract query method I mentioned above.

martin-gaievski · 2024-10-24T04:40:51Z

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

+        Query query = searchContext.query();
+        if (query instanceof BooleanQuery) {
+            BooleanQuery booleanQuery = (BooleanQuery) query;
+            hybridQuery = (HybridQuery) booleanQuery.clauses().get(0).getQuery();


this looks hacky, can we avoid this logic?

I have added the comment on top why it is written like that. Basically in case of nested fields and alias filter, hybrid query gets wrapped under bool query.

see my previous comment, this should be handled in here

src/main/java/org/opensearch/neuralsearch/search/query/HybridCollectorManager.java

Signed-off-by: Varun Jain <[email protected]>

* update maintainers.md Signed-off-by: Varun Jain <[email protected]> * update codeowners file Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]>

Signed-off-by: Varun Jain <[email protected]>

martin-gaievski · 2024-11-05T01:34:41Z

.github/CODEOWNERS

@@ -1,2 +1,2 @@
 # This should match the owning team set up in https://github.com/orgs/opensearch-project/teams
-*   @heemin32 @navneet1v @VijayanB @vamshin @jmazanec15 @naveentatikonda @junqiu-lei @martin-gaievski @sean-zheng-amazon @model-collapse @zane-neo @ylwu-amzn @jngz-es @vibrantvarun @zhichao-aws @yuye-aws
+*   @heemin32 @navneet1v @VijayanB @vamshin @jmazanec15 @naveentatikonda @junqiu-lei @martin-gaievski @sean-zheng-amazon @model-collapse @zane-neo @vibrantvarun @zhichao-aws @yuye-aws @minalsha


this looks like artifacts of improper rebase, can you please rebase on main properly

martin-gaievski · 2024-11-05T01:36:14Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

@@ -48,16 +50,23 @@ public final class HybridQueryBuilder extends AbstractQueryBuilder<HybridQueryBu
    public static final String NAME = "hybrid";

    private static final ParseField QUERIES_FIELD = new ParseField("queries");
+    private static final ParseField DEPTH_FIELD = new ParseField("pagination_depth");


Suggested change

private static final ParseField DEPTH_FIELD = new ParseField("pagination_depth");

private static final ParseField PAGINATION_DEPTH_FIELD = new ParseField("pagination_depth");

martin-gaievski · 2024-11-05T01:42:00Z

src/main/java/org/opensearch/neuralsearch/query/HybridQuery.java

 public final class HybridQuery extends Query implements Iterable<Query> {

    private final List<Query> subQueries;
+    private Integer paginationDepth;


why this is not primitive int? Operating with wrapper class is potentially error prone when boxing/unboxing a null value.

martin-gaievski · 2024-11-05T01:48:37Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

            querySearchResult.topDocs(updatedTopDocsAndMaxScore, querySearchResult.sortValueFormats());
        }
+
+        final int from = querySearchResults.get(0).from();


do we need final?

martin-gaievski · 2024-11-05T01:52:57Z

src/main/java/org/opensearch/neuralsearch/processor/NormalizationProcessorWorkflow.java

            querySearchResult.topDocs(updatedTopDocsAndMaxScore, querySearchResult.sortValueFormats());
        }
+
+        final int from = querySearchResults.get(0).from();
+        if (from > 0 && from > totalScoreDocsCount) {


first check looks redundant, can't we rely only on from > totalScoreDocsCount?

martin-gaievski · 2024-11-05T02:13:48Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryIT.java

@@ -637,6 +639,7 @@ public void testWrappedQueryWithFilter_whenIndexAliasHasFilterAndIndexWithNested

            HybridQueryBuilder hybridQueryBuilder = new HybridQueryBuilder();
            hybridQueryBuilder.add(QueryBuilders.existsQuery(TEST_TEXT_FIELD_NAME_1));
+            // hybridQueryBuilder.paginationDepth(10);


is this intentional?

martin-gaievski · 2024-11-05T02:17:57Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryIT.java

    @SneakyThrows
-    public void testHybridQuery_whenFromIsSetInSearchRequest_thenFail() {
+    public void testPaginationDepth_whenSubqueriesCountIsGreaterThanFive_thenFail() {


What are we testing here? check for max sub-query limit is part of the HybridQueryBuilderTests.

martin-gaievski · 2024-11-05T02:23:37Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

@@ -97,6 +109,9 @@ protected void doXContent(XContentBuilder builder, Params params) throws IOExcep
            queryBuilder.toXContent(builder, params);
        }
        builder.endArray();
+        if (isClusterOnOrAfterMinReqVersionForPaginationInHybridQuery()) {
+            builder.field(DEPTH_FIELD.getPreferredName(), paginationDepth == null ? DEFAULT_PAGINATION_DEPTH : paginationDepth);


when we're doing today's logic? from this statement we'll either get user provided value or default which is 10. Today's logic is equivalent to PD equals to 1, I'm not exactly sure, but definitely not 10.

martin-gaievski · 2024-11-05T02:27:06Z

src/main/java/org/opensearch/neuralsearch/query/HybridQueryBuilder.java

+        if (paginationDepth != null
+            && (paginationDepth < LOWER_BOUND_OF_PAGINATION_DEPTH || paginationDepth > UPPER_BOUND_OF_PAGINATION_DEPTH)) {
+            throw new IllegalArgumentException(
+                String.format(Locale.ROOT, "Pagination depth should lie in the range of 1-1000. Received: %s", paginationDepth)


please came up with better error message. The range is static here, but that's different in the logic where you have variables for lower and upper bounds, and we can drop the user-provided value as that's potentially unsafe, actual value should be obvious to user from the search request

martin-gaievski · 2024-11-05T02:29:53Z

src/test/java/org/opensearch/neuralsearch/query/HybridQueryIT.java

+    }
+
+    @SneakyThrows
+    public void testPaginationOnSingleShard_whenConcurrentSearchEnabled_thenSuccessful() {


Looks like you're basically put all tests for pagination to integ test. Can this be moved to unit tests, so we have only minimal increase in new integ tests. If that's not possible please consider moving them to a new class, this one because way overloaded, about 1000 lines.

vibrantvarun added 6 commits September 25, 2024 15:18

POC

5c9656b

Signed-off-by: Varun Jain <[email protected]>

Adding validation

58cd4c1

Signed-off-by: Varun Jain <[email protected]>

adding validation in depth

f1c64dc

Signed-off-by: Varun Jain <[email protected]>

Integ Test and unit test

152aa4c

Signed-off-by: Varun Jain <[email protected]>

Rebase with main

4d3e3d3

Signed-off-by: Varun Jain <[email protected]>

Update changelog

85edf35

Signed-off-by: Varun Jain <[email protected]>

vibrantvarun changed the title ~~Pagination in hybrid query~~ [Feature] Pagination in hybrid query Oct 23, 2024

martin-gaievski reviewed Oct 24, 2024

View reviewed changes

vibrantvarun added 7 commits October 25, 2024 15:13

Update maintainers.md (opensearch-project#966)

9d335ff

Signed-off-by: Varun Jain <[email protected]>

Add IT and Unit test for subqueries count in hybrid clause

085710a

Signed-off-by: Varun Jain <[email protected]>

Nominate Minal Shah (minalsha) as maintainer (opensearch-project#971)

180a5a6

* update maintainers.md Signed-off-by: Varun Jain <[email protected]> * update codeowners file Signed-off-by: Varun Jain <[email protected]> --------- Signed-off-by: Varun Jain <[email protected]>

refactoring code

e0ed9cb

Signed-off-by: Varun Jain <[email protected]>

Merge branch 'main' into Pagination_in_hybrid_query

0f8590b

Fixing Integ Tests

c5d3c46

Signed-off-by: Varun Jain <[email protected]>

Merge branch 'pagination_draft' into Pagination_in_hybrid_query

3a9344f

vibrantvarun marked this pull request as ready for review November 4, 2024 03:08

vibrantvarun requested review from heemin32, navneet1v, VijayanB, vamshin, jmazanec15, naveentatikonda, junqiu-lei, sean-zheng-amazon, model-collapse, zane-neo, zhichao-aws and yuye-aws as code owners November 4, 2024 03:08

martin-gaievski reviewed Nov 5, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Pagination in hybrid query #963

[Feature] Pagination in hybrid query #963

vibrantvarun commented Oct 23, 2024 •

edited

Loading

martin-gaievski Oct 24, 2024

vibrantvarun Nov 4, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Oct 24, 2024

vibrantvarun Nov 4, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

martin-gaievski Nov 5, 2024

	private static final ParseField DEPTH_FIELD = new ParseField("pagination_depth");
	private static final ParseField PAGINATION_DEPTH_FIELD = new ParseField("pagination_depth");

[Feature] Pagination in hybrid query #963

Are you sure you want to change the base?

[Feature] Pagination in hybrid query #963

Conversation

vibrantvarun commented Oct 23, 2024 • edited Loading

Description

Related Issues

Check List

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vibrantvarun commented Oct 23, 2024 •

edited

Loading