Explanation of what exactly the various dedup strategies do

[aditya-hari]

Is there any documentation of what the various deduplication strategies actually do? My use case is that I have a corpus of texts and the document have certain template texts that reoccur with slight variations to them. I don't know if any of the strategies here fit for that, and the code examples and whatever documentation there is don't help. For instance what does "only_dedup_in_index" mean for SentenceDedup?

Maybe it's just because I'm not very familiar with deduplication in general that I'm struggling with this, so would appreciate help.