Fix errors arising from kernels running on a separate extension host from other extensions

[jmcphers]

This change fixes some issues that arise when extensions that try to use R/Python sessions are running on a different extension host than the kernels themselves.

This fundamentally does not work today because sessions live in the extension host, and many of their methods are accessible only from that extension specific host or the main thread. It is not possible for one extension host to talk directly to another extension host.

Adding full support for this is more than I wanted to bite off, so this change is a bit of a stopgap. Here's how it works:

• API methods no longer return full-fledged session objects. Instead they return a more basic BaseLanguageRuntimeSession object.
• When an extension asks for a session object for a session that is not known to the current extension host, we generate a kind of proxy that lets us talk to the session via the main thread.
• In some extensions, we really and truly need a full session object. In those cases, we play a little bit dirty: we use package.json's extensionDependencies field to declare a dependency on the kernels and/or supervisor. When VS Code is deciding which extensions to run on which host, it ensures that extensions that depend on each other are on the same host -- so we're guaranteed that the sessions in question live on our own host. That allow us to safely cast the BaseLanguageRuntimeSession to a full LanguageRuntimeSessiion.

This makes a for a lot of running around to get e.g. a code result:

graph TD
e1[Extension 1 - Consumer] -- execute code --> eh1 
eh1[Ext Host 1] -- session proxy --> p[Positron]
p -- execute code --> eh2[Ext Host 2]
eh2 -- execute code --> e2[Extension 2 - Kernel]
e2 -- result --> eh2
eh2 -- result --> p
p -- result --> eh1
eh1 -- result --> e1

Loading

But it's about the best we can do given the architecture today.

Addresses #9166.

Release Notes

New Features

• N/A

Bug Fixes

• Fix errors when kernels are run on a separate extension host (Jupyter extension can run in different extension host than R/Python kernels #9166)

QA Notes

We don't actually know how Julia's system got into a state where it decided to put the kernels on a different extension host. But you can make yours do the same thing by adding this to your user settings JSON:

    "extensions.experimental.affinity": {
        "positron.positron-r": 1,
        "positron.positron-supervisor": 1
    }

It is possible that there are other problems in this configuration that I didn't fix. If adding the above setting gives you any trouble, check and see if it also repros on main. If it does, please file a separate issue; we think this config may be more common in the future.

[jmcphers]

Automated tests suggest some problems with LSPs? Marking as draft while I investigate.

[jmcphers]

OK, this is passing tests and ready for review. https://github.com/posit-dev/positron/actions/runs/17335967042 is a full suite run (felt necessary since this change is so low-level)

[juliasilge]

This is working great for me and honestly I do prefer dealing with the sessionId over searching through un-meaningful handle entries like we did before.

Working great, like I said, for all the real examples where I was running into errors:

I also checked with the settings to force the extension affinities. ✅

[juliasilge]

So Assistant really does have to run in the same process, but the Jupyter extension doesn't? Do you know why? (Just for my own learning)

[jmcphers]

Good question... it's because Assistant runs code with execution observers, which are one of the things that hasn't been factored to work across different extension hosts.