AI: Closing The Loop - Frontend

[jlewi]

A core hypothesis is that Runme lets us log the actions taken by developers and this can be used to continuously improve the model.

To support this we need an RPC service that

1. Lets us log notebooks
2. Instrument runmedev/web to log events.

We have some prior art from foyle

• LogEvents - RPC service to log events
• EventReporter Interface for reporting events from the Runme vscode extension

I think since we aren't constrained by the vscode-platform we have more flexibility to figure out how to handle logging.

At a high level what we'd like to do is periodically snapshot the notebook so the Agent on the backend can index it or otherwise process it to improve itself.

Additionally, since we are persisting notebooks in indexed storage we can potentially enqueue events and send them the next time the app is opened and the network is accessible.

@sourishkrout Do you have thoughts on what a simple, initial implementation for wiring up a LogEvents client?

Could we have a background process which checks which notebooks have changed since their last logging event and then sends an event to log them?

I think the schema is defined by SessionRecord. So I think we could just update that with a field like "lastLogged" to keep track of the last logged time.

[sourishkrout]

At a high level what we'd like to do is periodically snapshot the notebook so the Agent on the backend can index it or otherwise process it to improve itself.

We currently call saveState anywhere in the app, and the storage implementation ensures that excessive writes are debounced to save consumers from having to worry about syncing/throttling snapshotting.

It's possible to evolve saveState to take an event that further describes the user action, thereby increasing fidelity. We could batch up the event information and the most recent snapshot of the document as part of working the queue to keep the overhead low.

Additionally, since we are persisting notebooks in indexed storage we can potentially enqueue events and send them the next time the app is opened and the network is accessible.

@sourishkrout Do you have thoughts on what a simple, initial implementation for wiring up a LogEvents client?

It'd be possible to report snapshots of the document state to the agent asynchronously. Either directly or in a journaled way as part of tracking the persisted database state.

Could we have a background process which checks which notebooks have changed since their last logging event and then sends an event to log them?
I think the schema is defined by SessionRecord. So I think we could just update that with a field like "lastLogged" to keep track of the last logged time.

I'd suggest scheduling the reporting of the snapshots using additional db table columns as per your suggestion. We can debounce/synchronize this process to batch all unreported records instead of reporting them individually. The reporting can be scheduled for the initial session recovery and any subsequent time changes that occur.

If we want to absolutely minimize loss and latency, we could investigate whether it's possible to use a worker thread (the capabilities of which are unclear to me within the browser sandbox) to send event reports to the server. However, separating producers and the consumer would allow this to be a multi-stage project.

[jlewi]

Thanks.

If we want to absolutely minimize loss and latency

I'm not too concerned about latency. My initial thought is that latency O(10) minutes is pretty great and I imagine running periodically every 10 minutes isn't going to be much of a problem.