Japanese language

[aTan-aka-Xellos]

Hello!
Does anybody have experience with applying SymSpell (or edit distance approach in general) to CJK languages such as Japanese?
Since they have such challenges as:

• very short words (if measure by kanji)
• unsegmented text (no spaces)
• no proper, big dictionary available
• kana-to-kanji conversion may lead to a large edit distance

Right now looking for the approach for building the DidYouMean tool for the Japanese language and so far a lot of things hint that all tools that are based on 'edit distance', like SymSpell won't work.

[wolfgarbe]

You could ask https://github.com/AtheS21/SymspellCPP , who writes "Tested with English and Japanese and this version produced similar result to the original C# code".

A Japanese word frequency dictionary can be found here: https://github.com/hermitdave/FrequencyWords/tree/master/content/2018/ja
or here https://github.com/dataiku/dss-plugin-nlp-preparation/tree/main/resource/dictionaries

With symSpell.WordSegmentation you may transform unsegmented text (without spaces) into segmented text (spaces are inserted between words)