Clarification Needed: Implementing InQuest for Real-time Video Processing

[CookieDecide]

Hello InQuest team,

I am currently working on a project within a lab at TU Darmstadt where we are aiming to implement the InQuest pipeline with a live video stream as the input, rather than the precomputed CSV files typically used.

While adapting the code for real-time video processing, I encountered the following code segment:

InQuest/simulator.py

Lines 223 to 230 in b0ffe05

	for frame in range(query.start_frame, query.end_frame + 1, step):
	proxy_val = proxy.predict(frame)
	oracle_pred, oracle_matches_predicate = oracle.predict(frame)
	if oracle_matches_predicate:
	targets.append(oracle_pred)
	target_frames.append(frame)
	sampling_strategy.sample(proxy_val, oracle_pred, oracle_matches_predicate, frame)

My concern is that both proxy.predict and oracle.predict are being called in the same loop for each frame. This seems counterintuitive because it appears that the oracle, which is supposed to be more costly and accurate, is being invoked as often as the proxy. Wouldn't this approach negate the intended efficiency benefits of using a proxy model in conjunction with an oracle?

Could you please provide some guidance on the purpose of this structure and whether this is the intended usage pattern? Additionally, any advice on adapting this for real-time processing would be greatly appreciated.

Thank you for your assistance!

Best regards,
Julian Lauermann

[mdr223]

Hi Julian,

You are correct that in a production deployment you will not want to call the oracle on every frame. In this case, we are calling the oracle here in order to compute the groundtruth result for the purpose of running an experimental evaluation. The data structures targets and target_frames are only used later on to compute what the groundtruth result is, which allows us to measure InQuest's RMSE relative to the groundtruth.

Inside of sampling_strategy.sample() -- where the logic for InQuest's sampling resides -- you will see that we only use the oracle predictions for certain frames which are selected according to the reservoir sampling logic.

TL;DR: we only call oracle.predict(frame) on line 225 for the purposes of running an experimental evaluation, in a production system you would not do this.