Problem with custom word list

[ghost]

I have i test file with 60 words in it, and when i run toipe -f words -n 10 there will words that occurs multiple times , also the words that are at the end of the text file don't even show up at all [the words at the first line or two occurs the most]

[Samyak2]

Hey @SamDc73. Could you check that your word list satisfies these assumptions?

also the words that are at the end of the text file don't even show up at all [the words at the first line or two occurs the most]

There's no bias given to words that occur at the beginning, a random word is chosen uniformly. It's possible that those words at the end of the file are being skipped because they don't satisfy the assumptions.

[ghost]

could you please (if you have the time of course ) test it ?
here the word list that I'm using : https://pastebin.com/0vP3XMKC
the command that I'm running is "toipe -f Notes/typing/words -n 10",
just run couple of times , it's very obvious that couple of words occur more than one time, and the words at the end barely show up

[Samyak2]

@SamDc73 thank you for posting the word list!

First, the word list was not sorted alphabetically. I had to sort it myself using (it needs to be sorted according to the 3rd assumption in the list):

LC_COLLATE=en_US.UTF-8 sort -d -o /path/to/wordlist /path/to/wordlist

Then, some of the words in the file had trailing spaces. I removed them using:

sed --in-place 's/[[:space:]]\+$//' /path/to/wordlist

(you can use any text editor to do it too, but this automated).

Here's the fixed word list for reference.

absolute
another
appreciated
appreciated
available
because
beginner
behavior
believe
competitive
criteria
currently
decisions
default
dependencies
description
differences
discovered
duplicates
dynamically
engineering
environmental
everything
extensions
favorite
financially
focused
guideline
inflation
league
navigate
obvious
occurring
parse
population
preference
prescription
prompt
question
robotics
separate
slight
theoretically
variable
visualization

---

After fixing the file, I found a bug in toipe. I was using complete word lists (which had words for all 26 letters), so the code assumed that too. Because of this, a bias was introduced. I will be fixing this by using a better word selection algorithm that doesn't need require as many assumptions and isn't so hyper optimized :)

Thank you again for bringing this to my notice and sending the word list.