FFI Execution Plans that spawn threads panic

[timsaucer]

Describe the bug

If you have an FFI Execution Plan that spawns it's own tasks, it has no reactor and causes a panic.

To Reproduce

This is most easily demonstrated in datafusion-python. Just create a table provider that spawns a thread. I have examples on my computer but they are non-trivial (and it's probably me working on this issue).

Expected behavior

We should never cause a panic across the ffi boundary.

Additional context

No response

[alamb]

Seems like there are two problems:

1. The panic is going across FFI boundaries
2. The FFI code doesn't have a tokio rutime setup

Maybe we need to have some way to pass in a handle to the tokio runtime 🤔 Handle across the FFI boundaries

[timsaucer]

There is only one point where we have ffi calls that are async, currently. This happens in the record batch stream. So I think what needs to happen is that within the private data for the RBS we also include an Option<Arc<Runtime>> and if it exists, we enter the runtime during the async calls.

We do pass a context. I worry about a possible problem where we could have deadlocks if we have two different runtimes each doing a blocking call. The only place that is immediately jumping out to me as a potential problem spot is RecordBatchReceiverStreamBuilder, but I haven't had time to thoroughly look through this.

I think the immediate next thing to do is to build up a good reproducible test, ideally that I could include as a unit test.

[timsaucer]

Initial testing of passing in an Option<Arc<Runtime>> to the table provider and feeding it down through the execution plan and record batch stream was successful.