Revamp JanusGraphMultiQueryStrategy for better parent step usage [cql-tests] [tp-tests] #3783

porunov · 2023-05-20T02:37:00Z

This commit improves JanusGraphMultiQueryStrategy to better support multi-query compatible parent steps.

This commit brings better support for repeat step by introducing next itaration registration process.
Previously repeat children steps was getting traversers registered from the beginning of all outer repeat steps which
could result in duplicate or unnecesary retrievals for the first batch. Moreover, next iterations were not considered.
This commit changes the approach to bring different repeat step modes which can change the batches registration behaviour
to aacount only the closest repeat step, all repeat steps, all only starts of all repeat steps.
This commit adds support to almost all known TinkerPop Parent steps.
The exception is match step. We didn't have proper outter start registration for match step previously and now as well.

To not introduce a behavior change in multi-nested repeat steps the configuration option query.batch.repeat-step-mode is added with 3 modes available (closest_repeat_parent - mode which takes the closest repeat parent into account only, starts_only_of_all_repeat_parents - mode which takes all repeat parent steps but only starts into account (previous behavior), all_repeat_parents - same as starts_only_of_all_repeat_parents but takes repeat ends into account as well (new default behavior)).

The previous approach of inserting and finding multiQuery steps were split on 2 phases:

Insert multi-query step to appropriate location.
Find appropriate location.

Current approach is:

Insert multi-query step everywhere it might potentially be helpful.
Find appropriate location.

See performance improvement in this PR here.

Fixes #3733
Fixes #3735
Fixes #2996

Thank you for contributing to JanusGraph!

In order to streamline the review of the contribution we ask you
to ensure the following steps have been taken:

For all changes:

Is there an issue associated with this PR? Is it referenced in the commit message?
Does your PR body contain #xyz where xyz is the issue number you are trying to resolve?
Has your PR been rebased against the latest commit within the target branch (typically master)?
Is your initial contribution a single, squashed commit?

For code changes:

Have you written and/or updated unit tests to verify your changes?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE.txt file, including the main LICENSE.txt file in the root of this repository?
If applicable, have you updated the NOTICE.txt file, including the main NOTICE.txt file found in the root of this repository?

For documentation related changes:

Have you ensured that format looks appropriate for the output in which it is rendered?

porunov · 2023-05-27T05:24:09Z

Benchmark tests in this comment are now outdated due to CQL storage improvements and new development iterations in this PR. Please, see the new benchmark tests in the following comment.

Executed added benchmarks both for this PR and for master branch. The result are below.

Master branch:

Benchmark                                                      (fanoutFactor)  Mode  Cnt      Score     Error  Units
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex             100  avgt    5     78.099 ±   5.815  ms/op
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex             500  avgt    5   2027.809 ±  48.320  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                100  avgt    5   1655.942 ±  54.545  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                500  avgt    5  41937.395 ± 928.357  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps               100  avgt    5    819.768 ±  16.683  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps               500  avgt    5  20423.948 ± 269.510  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                       100  avgt    5     18.711 ±   2.841  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                       500  avgt    5    308.186 ±   7.184  ms/op
CQLMultiQueryBenchmark.getNames                                           100  avgt    5     96.689 ±   8.848  ms/op
CQLMultiQueryBenchmark.getNames                                           500  avgt    5   2540.583 ± 191.119  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                   100  avgt    5     96.305 ±   6.908  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                   500  avgt    5   2466.490 ±  91.254  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                       100  avgt    5     25.953 ±   1.640  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                       500  avgt    5    460.648 ±  27.039  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                       100  avgt    5     24.027 ±   2.691  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                       500  avgt    5    425.427 ±  36.108  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                         100  avgt    5     10.331 ±   0.944  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                         500  avgt    5    195.654 ±  24.556  ms/op
# Run complete. Total time: 00:22:51

This PR:

Benchmark                                                      (fanoutFactor)  Mode  Cnt     Score     Error  Units
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex             100  avgt    5    79.839 ±   1.505  ms/op
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex             500  avgt    5  2069.551 ±  11.601  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                100  avgt    5   172.839 ±   6.408  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                500  avgt    5  4471.290 ± 283.744  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps               100  avgt    5    91.972 ±   5.487  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps               500  avgt    5  2384.490 ± 125.744  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                       100  avgt    5     5.442 ±   0.597  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                       500  avgt    5    93.286 ±   4.392  ms/op
CQLMultiQueryBenchmark.getNames                                           100  avgt    5   101.769 ±   8.236  ms/op
CQLMultiQueryBenchmark.getNames                                           500  avgt    5  2586.406 ±  54.791  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                   100  avgt    5    98.194 ±   2.311  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                   500  avgt    5  2477.925 ± 119.731  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                       100  avgt    5    11.203 ±   1.394  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                       500  avgt    5   235.142 ±  17.712  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                       100  avgt    5     9.140 ±   0.661  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                       500  avgt    5   182.895 ±  10.338  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                         100  avgt    5    10.695 ±   1.994  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                         500  avgt    5   198.177 ±  30.259  ms/op
# Run complete. Total time: 00:16:53

There were no performance difference for tests: getAllElementsTraversedFromOuterVertex, getNames, getNeighborNames, and getVerticesWithDoubleUnion.
There is noticeable performance improvements for the next tests:

getElementsWithUsingEmitRepeatSteps - ~9.5 times better performance in this PR.
getElementsWithUsingRepeatUntilSteps - ~8.5 times better performance in this PR.
getIdToOutVerticesProjection - ~3.4 times better performance in this PR.
getVerticesFilteredByAndStep - ~2 times better performance in this PR.
getVerticesWithCoalesceUsage - ~2.3 times better performance in this PR.

Conclusion: As this PR adds support to a number of new parent steps, it's expected that new parent steps will gain better performance in multi-query cases. As repeat step is now registering vertices with the start of emit and until steps it improved performance significantly for cases when emit or until use multiQueriable start steps.

porunov · 2023-05-31T23:15:24Z

I made another development iteration and improved multi-nested repeat steps usage for some cases (added 3 different modes of running multi-nested repeat steps).
I also made configuration name changes so that configurations related to batch usage are now all under query.batch namespace (exception only is query.batch-property-prefetch which is going to be removed in the next PR in favor of a better has step configuration option).
To reflect the difference between previous and the new behavior of multi-nested repeat steps I added more tests (into JanusGraphTest.java) as well as added the benchmark test getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex.

The new benchmarks are below.

Master branch:

Benchmark                                                                            (fanoutFactor)  Mode  Cnt      Score      Error  Units
CQLMultiQueryBenchmark.getAdjacentVerticesLocalCounts                                           100  avgt    5    913.079 ±   95.302  ms/op
CQLMultiQueryBenchmark.getAdjacentVerticesLocalCounts                                           500  avgt    5  22396.761 ±  508.255  ms/op
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex                                   100  avgt    5     65.777 ±    5.554  ms/op
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex                                   500  avgt    5   1696.742 ±   66.152  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                                      100  avgt    5   1773.719 ±  130.207  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                                      500  avgt    5  44844.336 ± 2172.093  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps                                     100  avgt    5    966.557 ±   72.224  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps                                     500  avgt    5  23464.234 ±  890.497  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                                             100  avgt    5     22.069 ±    1.610  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                                             500  avgt    5    339.046 ±   19.012  ms/op
CQLMultiQueryBenchmark.getNames                                                                 100  avgt    5     67.366 ±    4.711  ms/op
CQLMultiQueryBenchmark.getNames                                                                 500  avgt    5   1754.979 ±   91.972  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                                         100  avgt    5     67.001 ±    4.207  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                                         500  avgt    5   1716.489 ±   50.094  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                                             100  avgt    5     28.635 ±    1.736  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                                             500  avgt    5    492.627 ±   38.755  ms/op
CQLMultiQueryBenchmark.getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex             100  avgt    5    980.735 ±  159.522  ms/op
CQLMultiQueryBenchmark.getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex             500  avgt    5  24170.311 ± 1573.206  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                                             100  avgt    5     26.980 ±    1.991  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                                             500  avgt    5    444.707 ±   46.764  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                                               100  avgt    5      9.605 ±    0.786  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                                               500  avgt    5    189.058 ±    7.875  ms/op

Current branch:

Benchmark                                                                            (fanoutFactor)  Mode  Cnt     Score     Error  Units
CQLMultiQueryBenchmark.getAdjacentVerticesLocalCounts                                           100  avgt    5    78.974 ±   3.557  ms/op
CQLMultiQueryBenchmark.getAdjacentVerticesLocalCounts                                           500  avgt    5  2045.282 ± 126.455  ms/op
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex                                   100  avgt    5    70.654 ±  12.948  ms/op
CQLMultiQueryBenchmark.getAllElementsTraversedFromOuterVertex                                   500  avgt    5  1706.734 ± 129.775  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                                      100  avgt    5   143.162 ±  10.757  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingEmitRepeatSteps                                      500  avgt    5  3669.181 ± 491.287  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps                                     100  avgt    5    87.608 ±   5.499  ms/op
CQLMultiQueryBenchmark.getElementsWithUsingRepeatUntilSteps                                     500  avgt    5  2267.824 ±  78.047  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                                             100  avgt    5     5.056 ±   0.369  ms/op
CQLMultiQueryBenchmark.getIdToOutVerticesProjection                                             500  avgt    5    88.035 ±   3.122  ms/op
CQLMultiQueryBenchmark.getNames                                                                 100  avgt    5    68.424 ±   3.998  ms/op
CQLMultiQueryBenchmark.getNames                                                                 500  avgt    5  1737.775 ±  56.719  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                                         100  avgt    5    70.409 ±   8.366  ms/op
CQLMultiQueryBenchmark.getNeighborNames                                                         500  avgt    5  1770.955 ±  49.131  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                                             100  avgt    5    10.702 ±   0.524  ms/op
CQLMultiQueryBenchmark.getVerticesFilteredByAndStep                                             500  avgt    5   233.268 ±  11.827  ms/op
CQLMultiQueryBenchmark.getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex             100  avgt    5   199.920 ±  22.572  ms/op
CQLMultiQueryBenchmark.getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex             500  avgt    5  5217.254 ± 311.079  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                                             100  avgt    5     8.600 ±   0.513  ms/op
CQLMultiQueryBenchmark.getVerticesWithCoalesceUsage                                             500  avgt    5   178.345 ±   6.817  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                                               100  avgt    5    10.378 ±   2.382  ms/op
CQLMultiQueryBenchmark.getVerticesWithDoubleUnion                                               500  avgt    5   199.045 ±  22.012  ms/op

Performance improvements are reflected here:

getAdjacentVerticesLocalCounts - ~11 times better performance.
getElementsWithUsingEmitRepeatSteps - ~12 times better performance.
getElementsWithUsingRepeatUntilSteps - 10.5 times better performance.
getIdToOutVerticesProjection - ~4 times better performance.
getVerticesFilteredByAndStep - ~2.3 times better performance.
getVerticesFromMultiNestedRepeatStepStartingFromSingleVertex - ~4.7 times better performance.
getVerticesWithCoalesceUsage - ~2.8 times better performance.

Please, in case anyone has enough capacity and would like to review this PR (or just have any concerns / comments / suggestions without committing for the full review), leave a commit in this PR until the start of the next week.
In case I don't receive any comments by the start of the next week, I'm going to merge this PR using lazy consensus.

cc @JanusGraph/committers

FlorianHockmann

Impressive work, not only on the implementation itself, including the benchmarks, but also on the documentation!

I have only reviewed the documentation as I won't have enough time to review the code changes here. My comments are mostly about stylistic issues.

docs/changelog.md

docs/configs/janusgraph-cfg.md

docs/operations/batch-processing.md

…-tests] [tp-tests] This commit improves JanusGraphMultiQueryStrategy to better support multi-query compatible parent steps. 1. This commit brings better support for `repeat` step by introducing next itaration registration process. Previously `repeat` children steps was getting traversers registered from the beginning of all outer repeat steps which could result in duplicate or unnecesary retrievals for the first batch. Moreover, next iterations were not considered. This commit changes the approach to bring different `repeat` step modes which can change the batches registration behaviour to aacount only the closest `repeat` step, all `repeat` steps, all only starts of all `repeat` steps. 2. This commit adds support to almost all known TinkerPop Parent steps. The exception is `match` step. We didn't have proper outter start registration for `match` step previously and now as well. Fixes JanusGraph#3733 Fixes JanusGraph#3735 Fixes JanusGraph#2996 Signed-off-by: Oleksandr Porunov <[email protected]>

porunov · 2023-06-02T17:07:10Z

Impressive work, not only on the implementation itself, including the benchmarks, but also on the documentation!

I have only reviewed the documentation as I won't have enough time to review the code changes here. My comments are mostly about stylistic issues.

Thank you @FlorianHockmann ! I agree with all your comments and applied the changes you suggested.

FlorianHockmann

LGTM and it makes sense to address the Changelog in a follow-up issue.

porunov added kind/performance area/core labels May 20, 2023

porunov added this to the Release v1.0.0 milestone May 20, 2023

janusgraph-bot added the cla: external Externally-managed CLA label May 20, 2023

porunov force-pushed the feature/multi-query-strategy-revamp branch from 17b3485 to 464f027 Compare May 20, 2023 09:13

This was referenced May 20, 2023

Apply Repeat iteration result to multi-query Repeat beginning #3779

Closed

Use And Or steps multi-query for children MultiQueriable Steps #3736

Closed

porunov force-pushed the feature/multi-query-strategy-revamp branch 8 times, most recently from e73a570 to 052cfa7 Compare May 27, 2023 04:44

porunov marked this pull request as ready for review May 27, 2023 05:24

porunov requested a review from a team May 27, 2023 05:26

porunov force-pushed the feature/multi-query-strategy-revamp branch 3 times, most recently from d4688a8 to 42fd43d Compare May 31, 2023 12:33

porunov changed the title ~~Revamp JanusGraphMultiQueryStrategy for better parent step usage~~ Revamp JanusGraphMultiQueryStrategy for better parent step usage [cql-tests] [tp-tests] May 31, 2023

porunov force-pushed the feature/multi-query-strategy-revamp branch 2 times, most recently from 243f328 to ccec0b8 Compare May 31, 2023 16:52

porunov force-pushed the feature/multi-query-strategy-revamp branch from ccec0b8 to 7f21c5a Compare June 2, 2023 00:30

FlorianHockmann reviewed Jun 2, 2023

View reviewed changes

porunov force-pushed the feature/multi-query-strategy-revamp branch from 7f21c5a to 234bf71 Compare June 2, 2023 17:01

porunov force-pushed the feature/multi-query-strategy-revamp branch from 234bf71 to 89f5611 Compare June 2, 2023 17:05

porunov mentioned this pull request Jun 4, 2023

Improve the changelog and / or create a blog post about recent repeat step multi-query improvements #3799

Open

FlorianHockmann approved these changes Jun 5, 2023

View reviewed changes

porunov merged commit 6e73ef4 into JanusGraph:master Jun 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revamp JanusGraphMultiQueryStrategy for better parent step usage [cql-tests] [tp-tests] #3783

Revamp JanusGraphMultiQueryStrategy for better parent step usage [cql-tests] [tp-tests] #3783

porunov commented May 20, 2023 •

edited

Loading

porunov commented May 27, 2023 •

edited

Loading

porunov commented May 31, 2023 •

edited

Loading

FlorianHockmann left a comment

porunov commented Jun 2, 2023

FlorianHockmann left a comment

Revamp JanusGraphMultiQueryStrategy for better parent step usage [cql-tests] [tp-tests] #3783

Revamp JanusGraphMultiQueryStrategy for better parent step usage [cql-tests] [tp-tests] #3783

Conversation

porunov commented May 20, 2023 • edited Loading

For all changes:

For code changes:

For documentation related changes:

porunov commented May 27, 2023 • edited Loading

porunov commented May 31, 2023 • edited Loading

FlorianHockmann left a comment

Choose a reason for hiding this comment

porunov commented Jun 2, 2023

FlorianHockmann left a comment

Choose a reason for hiding this comment

porunov commented May 20, 2023 •

edited

Loading

porunov commented May 27, 2023 •

edited

Loading

porunov commented May 31, 2023 •

edited

Loading