Databricks Connect

[jrberg772]

Is it possible, or on the roadmap to connect to databricks in positron? Simmilar to how one can using databricks connect in vscode?

The databricks connect extension is great at enabling user to create and run code for databricks, however the data display/review features are lacking. It would be cool to see the compute of databricks combined with the data review features of positron, if at all possible.

[jrberg772]

To add some more details, I did try running some code myself. I downloaded the databricks extension. However, it does not seem to perform the same as it does on vscode. For example, not being able to recognize databricks cells in a .py file and running that code in the ide.

[jthomasmock]

We are interested in eventually supporting remote data in the Data Explorer, but if you want to open a specific feature request as an Issue, please go ahead!

I was able to run an example file remotely and get data back, but I've shared this thread with my contacts there.

[jthomasmock]

We do expect remote connections to Databricks via Databricks Connect to work in R sparklyr or Python pyspark

We also expect that the Databricks extension will work in Positron: https://www.open-vsx.org/extension/databricks/databricks

If you are running into issues, there may be some adjustments needed in their extension to make use of our new components. I'll reach out to our contacts over there. Would you be able to share some screenshots or additional information?

[jrberg772]

Hey @jthomasmock , thanks for the response. I may add that feature request.

One of the first things I noticed when running databricks connect in VScode and Positoron is that Positron seems to have less features present than VScode in regards to databricks connect.

1. Run code as cells

In vscode, it is possible to run databricks code in cells, which is a useful feature as one is building code. Especially for native databricks users that are familiar with the databricks notebook format.

2. Some built-in databricks features/assumptions are not present.

For example, I did not have to define spark. My databricks runtime already knows what spark is, databricks connect handles that core assumption (in vscode), along with other databricks features like dbutils.

So out of the box, databricks connect in vscode is easy to run code with the same/simmilar assumptions present if one were to write code in databricks itself. So some of my current repos and projects would need some modification to perform any development in positron. And I personally don't want to part ways with the in-line cell features in databricks in vscode.

[salexir]

Hey @jrberg772, @jthomasmock

I think the issue has to do with where the extensions are being sourced. The Positron marketplace uses OpenVSX and the latest Databricks version available there is v1.4.0, while VS Codes sources from VS Code Marketplace, which has version v2.4.8. I've been able to install v2.4.8 in Positron from the downloaded .vsix file, and the features map fully between Positron/VS Code afterwards.

[jthomasmock]

Thanks @salexir ! After talking with the Databricks team, I expect that the "new 2.0" version of Databricks extension will be up on OpenVSX by the end of the 2024.

[jthomasmock]

I have confirmed with the Databricks team and OpenVSX that the "2.0" extension is now up!

https://open-vsx.org/extension/databricks/databricks