Design goal of HPyCapsule conflicts with universal mode

[mattip]

When trying to implement the HPyCapsule interfaces for PyPY, I see the design goal is

HPyCapsule should be interchangeable with PyCapsule

How will that play out when we want to use universal mode? The struct is defined as

typedef struct {
    PyObject_HEAD   # which is PyObject ob_base;
    void *pointer;
    const char *name;
    void *context;
    PyCapsule_Destructor destructor;
} PyCapsule;

so it needs a CPython PyObject.

[mattip]

I think we should review this design goal, and HPyCapsule should be a pure-c struct with no PyObject_HEAD. This could be part of #446

[fangerer]

How will that play out when we want to use universal mode?

In HPy, we don't have a type HPyCapsule but what you can do is: in hybrid ABI, you can do something like this:

HPy h_capsule = HPyCapsule_New(ctx, ptr, "name", destructor);
PyCapsule *py_capsule = (PyCapsule *)HPy_AsPyObject(ctx, h_capsule);

This is basically what I meant with:

HPyCapsule should be interchangeable with PyCapsule

Further, the goal was that you can create a capsule object in a C API extension using PyCapsule_New, then store it to some Python attribute or whatever (like datetime.datetime_CAPI), and then you can retrieve it in an HPy extension and use HPyCapsule_* functions on it.

I think we should review this design goal

IMO, the goal is fulfilled because in HPy, we are not exposing internal of the capsule object. So, we can implement it as we want and in case of CPython, we implement it as PyCapsule.

What problems are you running into?

[mattip]

What problems are you running into?

None so far, just thinking about how to implement this in PyPy. It seems we should have a HPyCapsule which is

typedef struct {
    void *pointer;
    const char *name;
    void *context;
    HPyCapsule_Destructor destructor;
} HPyCapsule;

[mattip]

Tests for HPyCapsule are found in non-hybrid test_hpycapsule. How does this work with universal mode? Sorry, HPyCapsule can be implemented without using PyCapsule, as described above.

Also it seems HPy_SetCallFunction can only be used in hybrid mode, but is tested in universal mode.

[fangerer]

Also it seems HPy_SetCallFunction can only be used in hybrid mode, but is tested in universal mode.

Why do you think that HPy_SetCallFunction can only be used in hybrid mode? This would only be the case if there would be CPython-specific types in the signature but there aren't any. The signature is:

int HPy_SetCallFunction(HPyContext *ctx, HPy h, HPyCallFunction *func)

Type HPyCallFunction is defined in hpy.h and can be seen as a minimal HPyDef. It is defined as:

typedef struct {
    cpy_vectorcallfunc cpy_trampoline;
    HPyFunc_keywords impl;
} HPyCallFunction;

For type cpy_vectorcallfunc we do the same trick as for other function pointer types: we define this type in cpy_types.h using a signature that is compatible to CPython's vectorcallfunc. In the signature, we use cpy_PyObject * instead of PyObject *.

Anyway, are you running into any problem implementing this function?
I didn't have any problem implementing it on GraalPy. We just ignore field cpy_trampoline and only read field impl.

[mattip]

OK, I will try.

[mattip]

And you assign it to the object's __call__ slot, overriding any possibly existing slot value? PyPy does not really have a tp_vectorcall slot on its objects.

[fangerer]

Yes, on GraalPy, we install a special built-in function to attribute __call__ that will read the function pointer from the object. We always assume that the object-specific function pointer is properly initialized by the constructor (i.e. by the HPy_tp_init function). If the constructor is inherited (e.g. if the HPy type inherits from Python int or from a other user-defined type), we decorate the inherited constructor and set the call function pointer (using the default call function from the type). If the user provides custom constructor, this one needs to use HPy_SetCallFunction to do the initialization.

[mattip]

Closing, thanks for the help.