-
Notifications
You must be signed in to change notification settings - Fork 121
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problem with diacritics and transliterating to lists #174
Comments
Ok, for anyone facing this same issue, I have written a solution for postprocessing the delimited strings:
Now if you run the following code the delimited string is correctly split:
This outputs:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I've run into some issues in several languages, where diacritics lead to strange behavior.
Example in French:
which yields the outputs
when using space as a delimiter the diacritic attaches itself to the next letter, when using any other delimiter like tilder, it outputs an extra delimiter and the diacritic then modifies the delimiter (in this case a tilder, but the same happens with any chosen delimiter).
This happens in other languages as well, so far I've tried Portuguese, Italian, same thing.
Is this expected behavior or is there some kind of trick I am unaware of? To my understanding a diacritic is not considered an additional phoneme, but instead a modifier. I also understand that unicode uses a postfix notation for diacritics, so is this perhaps an encoding issue?
The text was updated successfully, but these errors were encountered: