How to get a context when python is embedded in an application?

[KubaO]

My use case is not module development, but having a (currently) CPython interpreter embedded in the application. I'm using HPy so that in distant future a different Python implementation could be embedded.

How do I get the context to actually do something with HPy? Is _HPyGetContext the way to do it?

I've seen #268 and it'd seem to imply that there's no such API. That's... wierd?

[fangerer]

How do I get the context to actually do something with HPy?

The HPyContext is HPy's ABI and so always provided by the Python interpreter. Currently, the only way to get one is to write an HPy extension with methods/slots/etc. and use those (i.e. call those) from Python.
As you noticed, we discussed this topic already but mainly for the use case where an HPy extension uses some generic library (e.g. some compression library). An API to fetch the HPyContext would make sense if the library would have a callback mechanism and if this callback function would then again want to use HPy API. In this case it is valid because there are already functions on the stack that received an HPyContext from the runtime but you just don't have any possibility to pass the context through.

However, we will probably never provide an API that would serve as an entry point for Python. I mean, if you write a little C program that links to libpython (assume with HPy) and you want to directly call some HPy function without going through the Python engine. This would mean that the API needs to initialize the whole Python runtime and this doesn't make sense for HPy. In this case, you would need to initialize your Python runtime first (using some runtime-specific C API) and then eval some Python code that calls an HPy extension.

Is _HPyGetContext the way to do it?

No, definitively not. This is an implementation detail (only on CPython). Other Python runtimes like GraalPy and PyPy won't have this variable.

[hodgestar]

@KubaO Could you give an example of how you would like to use the API. So you have an embedded interpreter running. That interpreter can already itself load HPy modules. What are the other situations in which you would like to call C functions that call the embedded Python's C API?

I can imagine that an embedded Python might provide a function that calls a callback and provides a context to it, or some other means to run functions with a context to interact with the embedded interpreter.

[KubaO]

There's quite a bit of code in KiCad that calls "into" Python interpreter from code that itself was not invoked from Python code.

What are the other situations in which you would like to call C functions that call the embedded Python's C API?

C++ UI code calls into Python from various points, to invoke user scripts etc. It's a common pattern where Python is used for scripting. Pretty much every project that has a UI and uses Python for scripting would be doing that.

[hodgestar]

@KubaO Thanks for the explanation.

Perhaps for the embedding case we could provide a method that returns an opaque reference to the interpreter, and another method that can use that reference to request a temporary execution context.

E.g.

HPyInterp interpreter;
interpreter = HPy_Initialize();
ctx = HPy_EnterContext(interpreter);
... do things with the HPy API ...
HPy_ReleaseContext(interpreter);

The intention is that this would be very similar to executing a C extensions function from the interpreter. The section between HPy_EnterContext and HPy_ReleaseContext should not be too crazy, all handles should be closed, debug mode should check things on HPy_ReleaseContext, etc.

CFFI added some support for embedding Python without directly exposing the C API -- https://cffi.readthedocs.io/en/latest/embedding.html. That might be a bit painful for porting though? It's more helpful for executing Python code than for calling the C API directly from outside the interpreter.