Component for highlighting chunks based on the generated response/answer

[NISH1001]

Currently because of generative nature of the search engines and RAG components in larch, there's no way to know which part of the chunks the response has been generated.

I propose a new feature component like ChunkHighlighter or something that takes in list of chunks and generated response (all being texts/strs) and have downstream implement some highlighting mechanism.

For example, here's the basic token-based highlighter we can have

from larch.highlighters import ChunkHighlighter

class TextSpan(BaseModel):
    start: Optional[int] = None
    end: Optional[int] = None
    text: Optional[str] = None

class TokenBasedHighlighter(ChunkHighlighter):
    def compute_highliights(self, chunks: List[str], text: str) -> List[Tuple[TextSpan]]:
        tokens = text.split()
        highlights = []
        for chunk in chunks:
            chunk_highlights = []
            for token in tokens:
                start = chunk.find(token)
                if start != -1:
                    chunk_highlights.append(TextSpan(start=start, end=start+len(token)))
            highlights.append(chunk_highlights)
        return highlights


text = "this is a dummy response"

chunks = [
"There are multiple ways to generate dummy things as response",
"What a dummy chunk this is"
]

highlights = TokenBasedHighlighter()(chunks=chunks, text=text)

# maybe we could even have greedy approach, that gets longest highlight in the index if they are sequential in the text chunk
highlights = TokenBasedHighlighter(greedy=True)(chunks=chunks, text=text)