Spellchecking should be able to return multiple suggestions

[kelson42]

If the best result shares the same Levenstein (or similar) distance

Considering Xapian can not do that out-of-the-box, this issue depends on #1014

[kelson42]

@veloman-yunkan I believe this is the most prioritary feature to implement now in the spellchecker. You told me you have a way to do it, but this is a bit hacky, but the following things are unclear so far:

• What is the nature of the problem
• How the solution would look like

I think it would be better to be clear about the this two points above before implementing a PR.

[veloman-yunkan]

@kelson42

Current implementation of spelling correction (which, by the way, is fully contained in libkiwix) relies on xapian's spelling correction functionality. The latter only supports a single spelling correction per query. That limitation can be worked around by temporarily removing the returned correction from the spelling database and repeating the request, whereupon the next best correction will be returned. That procedure can be performed as many times as needed to obtain the desired number of corrections. The removed entries then must be re-inserted into the spellings DB. Such a hack has the following drawbacks:

1. Spelling correction is not supported by the in-memory backend of Xapian, i.e. the glass (on-disk) backend has to be used, and the spelling correction operation thus becomes a non-readonly operation with respect to the on-disk data leading to:
   a. extra wear of SSD storage
   b. risk of data corruption (loss of spelling entries) if the application crashes during the spelling correction function call (this can be worked around with additional measures)
   c. spelling correction cannot be called concurrently in kiwix-serve
   d. slow

Besides, Xapian's algorithm for spelling correction is based on edit distance, rather than on phonetic similarity. If we intend to eventually provide real spellchecking instead of a surrogate one, we should use a real spellchecking engine. Switching to one will automatically enable multiple corrections.

[kelson42]

@veloman-yunkan Thx for the explanation. My conclusion is that this issue is blocked by #1014 and we should implement #1014 first.

[ojwb]

b. risk of data corruption (loss of spelling entries) if the application crashes during the spelling correction function call (this can be worked around with additional measures)

(As I noted a few minutes ago in kiwix/libkiwix#1254 (comment)) this at least could be avoided by calling impl_->begin_transaction(); before you start removing spellings, and then you can rollback the temporary changes with impl_->cancel_transaction();. Even if the database is explicitly closed during error handling, the transaction gets implicitly cancelled.

You probably wouldn't actually end up with the changes implicitly committed anyway (because auto commit is based on the number of document changes and spelling changes aren't counted), but putting it in a transaction you then cancel guarantees it won't get committed and should require less work too.

[veloman-yunkan]

@ojwb Noted, thanks!

[ojwb]

There was actually a GSoC project to implement various improvement to Xapian's spelling correction back in 2011:

https://trac.xapian.org/wiki/GSoC2011/SpellingCorrectionImprovements

My memory is it was a bit of a frustrating one - the student didn't really take guidance and just ploughed on churning out code, and the result was a big pile of intermingled changes rather than a series of clean patches each doing one thing.

Dan (the main mentor) tried to peel off useful parts and clean them up to merge, but I think got disheartened. It didn't help that the first part of the work was a new way to calculate the edit distance candidates which claimed to deliver impressive performance which we just couldn't reproduce.

It might still be possible to pull out particular things from it. The student seemed bright and able, we just failed to steer them well enough.

Using a different dedicated spelling correction library may make more sense for you though. The really key thing using Xapian's spelling correction allows which is probably harder to achieve using a dedicated library is dynamic spelling based (wholly or partly) on words in the corpus, but if I follow it seems you're just loading an existing dictionary.