Skip to content

Commit

Permalink
docs: fix incorrect reference to LLM in NonLLMContextPrecisionWithRef…
Browse files Browse the repository at this point in the history
…erence metric (#1863)

Updated the documentation for NonLLMContextPrecisionWithReference to
clarify that it uses non-LLM-based similarity measures, correcting the
misleading reference to LLMs.

fixes #1861
  • Loading branch information
sahusiddharth authored Jan 21, 2025
1 parent 064ceed commit 9da919e
Show file tree
Hide file tree
Showing 3 changed files with 4 additions and 4 deletions.
4 changes: 2 additions & 2 deletions docs/concepts/metrics/available_metrics/context_precision.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,11 +68,11 @@ Output

## Non LLM Based Context Precision

The following metrics uses traditional methods to identify if a retrieved context is relevant or not. You can use any non LLM based metrics as distance measure to identify if a retrieved context is relevant or not.
This metric uses traditional methods to determine whether a retrieved context is relevant. It relies on non-LLM-based metrics as a distance measure to evaluate the relevance of retrieved contexts.

### Context Precision with reference contexts

`NonLLMContextPrecisionWithReference` metric is can be used when you have both retrieved contexts and also reference contexts associated with a `user_input`. To estimate if a retrieved contexts is relevant or not this method uses the LLM to compare each of the retrieved context or chunk present in `retrieved_contexts` with each ones present in `reference_contexts`.
The `NonLLMContextPrecisionWithReference` metric is designed for scenarios where both retrieved contexts and reference contexts are available for a `user_input`. To determine if a retrieved context is relevant, this method compares each retrieved context or chunk in `retrieved_context`s with every context in `reference_contexts` using a non-LLM-based similarity measure.

#### Example

Expand Down
2 changes: 1 addition & 1 deletion src/ragas/testset/synthesizers/multi_hop/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ async def _generate_sample(
self, scenario: Scenario, callbacks: Callbacks
) -> SingleTurnSample:
if not isinstance(scenario, MultiHopScenario):
raise TypeError('scenario type should be MultiHopScenario')
raise TypeError("scenario type should be MultiHopScenario")
reference_context = self.make_contexts(scenario)
prompt_input = QueryConditions(
persona=scenario.persona,
Expand Down
2 changes: 1 addition & 1 deletion src/ragas/testset/synthesizers/single_hop/base.py
Original file line number Diff line number Diff line change
Expand Up @@ -122,7 +122,7 @@ async def _generate_sample(
self, scenario: Scenario, callbacks: Callbacks
) -> SingleTurnSample:
if not isinstance(scenario, SingleHopScenario):
raise TypeError('scenario type should be SingleHopScenario')
raise TypeError("scenario type should be SingleHopScenario")
reference_context = scenario.nodes[0].properties.get("page_content", "")
prompt_input = QueryCondition(
persona=scenario.persona,
Expand Down

0 comments on commit 9da919e

Please sign in to comment.