XWIKI-15166: SOLR reindexation job cause serious perfomance issues on large database #4536
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… large database * missing proper ordering in the diff iterator when there are holes
michitux
requested changes
Sep 3, 2025
...m-search-solr-api/src/main/java/org/xwiki/search/solr/internal/job/SolrDocumentIterator.java
Show resolved
Hide resolved
… large database * better variable name
… large database * missing field in the solr query
… large database * update integration test
It was a bit of a pain to build integration tests because of the state of nexus-snapshot yesterday. I should be OK now (it's passing at least), but I could use your input on the change I made to make it pass (the main one being that we don't compute any comparison anymore in #toList for the diff iterator, since we don't have the docId). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
@michitux noticed a problem in how the ordering of missing entries was handled in the diff iterator from #4298, so here is an attempt to fix that
Jira URL
https://jira.xwiki.org/browse/XWIKI-15166
Changes
Description
Since we need the docId to properly compare entries, I had to change what the iterators expose to also provide the docId.
Clarifications
DocumentIteratorEntry#wiki:I could have used the wiki reference from the DocumentReference, but I wanted to be able to sort with only the DocumentIteratorEntry to make the code simpler.
Executed Tests
Mainly updated already existing tests.
Expected merging strategy