Input already tokenized array

[Nireas1]

Hello,
Is it possible to use an already tokenized text as input?
Right now from my understanding we have to have a string and we can get the tokens like below:
const doc = nlp.readDoc(text); // where text is of type string
const tokens = doc.tokens().out();

The text I have is already tokenized as an array of strings and although I can use the 'as' helper wrongly eg. like this as.bow(alreadyTokenizedText) to get a bag of words, there are a lot of limitations with its other functions or the 'its' helper.
I might be able to use the wink-nlp-utils but as far as I saw it doesn't have functionality like bm25 or similarity detection methods that I want.
So is there a way to use an array of strings as input instead of text or any other solution?

Thank you for your time.

[rachnachakraborty]

Hi @Nireas1,

We are not very clear about the exact problem you are experiencing using winkNLP.

BM25 Vectorization and Similarity features are very much available in winkNLP taking tokenized text as an input. Please refer to the documentation .

Request you to share the use case along with the problem statement.

Thanks,
Rachna

[Nireas1]

Sorry if I wasn't clear.
What I meant is that there are a lot of limitations when having tokenized text as input because you can't use things like the helpers.
In order to use them you have to go through the readDoc method that only takes string.

For example, if I wanted to remove the stopwords from an already tokenized text like below:
const t = ["this", "is", "an", "example"];
I wouldn't be able to do it since I can't use the its.stopWordFlag.

Is my only choice to use wink-nlp in conjuction with other packages like wink-nlp-utils to solve that limitation?

Again, thank you for your time.

[rachnachakraborty]

Hi,

Thanks for elaborating on the requirement.

Your need is unique for winkNLP as it is a processing pipe which is handling all the annotations with readDoc('input text'). These annotations also include the token properties such as stop word, abbreviations, token type, negation and many more. Thus, an only array of tokens is not sufficient to use helpers.

You will have to follow a hybrid approach in such a case, like you already have tried.

Here are some how-tos on winkNLP.

Cheers,
Rachna

[Nireas1]

Thank you very much.
I just wanted to make sure I wasn't missing something.
I will use it in conjuction with other wink packages as you suggested, too.

Thanks again.

[rachnachakraborty]

Was curious to know, whether the input text is already tokenized and there is no access to the raw text?

In case of tokenized array, you can join them on space to create a raw text file and pass it through the winkNLP pipe and leverage all the features.

Please checkout if this option works for you.

Thanks,
Rachna

[Nireas1]

I tried that but the array is produced by wordnet and has some words like "particular(a)", "designate(ip)", "unique(p)" which nlp seperates into two different words with punctuation (eg. "designate", "(", "p", ")" ) and I would like to have them as is, because they seem to produce better results using similarity methods like cosine similarity.

Thanks,
Nireas