Use spaCy tokenizer for Dutch

[BramVanroy]

Dutch a relatively hard language to tokenize, especially when it comes to the possessive. From my testing in the past, I do prefer spaCy's tokenization though, as it does not have the stronger tendency (like NLTK) to split off an apostrophe when it is part of the possessive.

from datatrove.utils.word_tokenizers import SpaCyTokenizer, NLTKTokenizer

tokenizer = SpaCyTokenizer("nl")

print(tokenizer.word_tokenize("Ik eet die graag 's morgens. 's Anderdaags zie ik oma's kippen in Belgiës troeven. Dante’s hel en Louis’ honden."))
# ['Ik', 'eet', 'die', 'graag', "'s", 'morgens', '.', "'s", 'Anderdaags', 'zie', 'ik', "oma's", 'kippen', 'in', 'Belgiës', 'troeven', '.', 'Dante', '’s', 'hel', 'en', 'Louis', '’', 'honden', '.']


tokenizer = NLTKTokenizer("dutch")
print(tokenizer.word_tokenize("Ik eet die graag 's morgens. 's Anderdaags zie ik oma's kippen in Belgiës troeven. Dante’s hel en Louis’ honden."))
# ['Ik', 'eet', 'die', 'graag', "'s", 'morgens', '.', "'s", 'Anderdaags', 'zie', 'ik', 'oma', "'s", 'kippen', 'in', 'Belgiës', 'troeven', '.', 'Dante', '’', 's', 'hel', 'en', 'Louis', '’', 'honden', '.']

Neither of these are perfect but I prefer the spaCy one.

[guipenedo]

Thank you for the insight! We have been experimenting a bit and had actually already started swapping out nltk based tokenizers for spaCy, but it's good to have some confirmation from someone who speaks one of the affected languages.

[BramVanroy]

Great, let me know if you need me to look at other things specific to Dutch!

[BramVanroy]

Any chance we can get this merged @guipenedo?