Feature request: Return both summary & eval_results from Evaluation.evaluate

[TeoZosa]

For this code:

weave/weave/flow/eval.py

Lines 493 to 510 in 3eea8df

	@weave.op()
	async def evaluate(self, model: Union[Callable, Model]) -> dict:
	# The need for this pattern is quite unfortunate and highlights a gap in our
	# data model. As a user, I just want to pass a list of data `eval_rows` to
	# summarize. Under the hood, Weave should choose the appropriate storage
	# format (in this case `Table`) and serialize it that way. Right now, it is
	# just a huge list of dicts. The fact that "as a user" I need to construct
	# `weave.Table` at all is a leaky abstraction. Moreover, the need to
	# construct `EvaluationResults` just so that tracing and the UI works is
	# also bad. In the near-term, this will at least solve the problem of
	# breaking summarization with big datasets, but this is not the correct
	# long-term solution.
	eval_results = await self.get_eval_results(model)
	summary = await self.summarize(eval_results)
	print("Evaluation summary", summary)
	return summary

Only summary is returned. For our use-case, we also want to grab eval_results for downstream rendering and storage (to present user-friendly results to non-technical stakeholders; the Weave UI is information overload for those folks). Would this change make sense?

As a workaround, I can call get_eval_results and summarize separately, but lose eval tracking in Evaluations since only Evaluation.evaluate Calls are picked up.

[jwlee64]

Hi @TeoZosa, I can raise this to the team tomorrow.

[TeoZosa]

@jwlee64 sounds good, thanks for the prompt reply 👍

FWIW for feedback, I'm working around the issue by vendoring the code with this change:

-   @weave.op()
+   @weave.op(postprocess_output=lambda output: output[0])
    async def evaluate(self, model: Union[Callable, Model]) -> dict:
        eval_results = await self.get_eval_results(model)
        summary = await self.summarize(eval_results)

        print("Evaluation summary", summary)

-        return summary
+        return summary, eval_results

[jwlee64]

Hi @TeoZosa, we are going to have someone tackle this in the next week or so, or at minimum document a better way to get the eval_results with the current api.

[TeoZosa]

Got it. Thanks for the update @jwlee64, keep me posted! 🙏