chore: upgrade to datafusion 43

[ion-elgreco]

Description

Bump kernel

• closes Schema metadata descriptions cause write failures in deltalake >= 0.19.0 #2850

[rtyler]

We need Datafusion to upgrade as well before we can do this. I have some patches to make the latest datafusion work but we're blocked on them to adopt the newest arrow and kernel

[Xuanwo]

Hi, datafusion has been released. Let's rock!

[alamb]

Thank you @ion-elgreco -- let me know if there is anything I can to do help with this PR (e.g. I could work on updating the code to use non deprecated APIs in the core for parquet statistics...)

[ion-elgreco]

Hey @alamb, I am not working on this atm, but @rtyler has picked it up from here

[ion-elgreco]

@alamb there seems to be a regression in DF, see https://github.com/delta-io/delta-rs/actions/runs/10959345721/job/30431504526?pr=2886#step:7:948.

A literal with a list is creating a dtype list with inner type using the name "item", we however have our data internally read according to the parquet spec, so that when we write the parquet files are meeting the field naming of the spec.

I had a quick glance at the Changelog but can't really pinpoint where this change may have occurred, do you have any ideas?

DFSchema { inner: 
Schema { fields: [
Field { name: "items_in_bucket", data_type: List(Field { name: "element", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }, 
new schema: 
DFSchema { inner: Schema { fields: [
Field { name: "items_in_bucket", data_type: List(Field { name: "item", data_type: Utf8, nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} }), nullable: true, dict_id: 0, dict_is_ordered: false, metadata: {} },

[alamb]

@alamb there seems to be a regression in DF, see https://github.com/delta-io/delta-rs/actions/runs/10959345721/job/30431504526?pr=2886#step:7:948.

A literal with a list is creating a dtype list with inner type using the name "item", we however have our data internally read according to the parquet spec, so that when we write the parquet files are meeting the field naming of the spec.

It sounds somewhat similar to a discussion happening here about nullability (but I think field name is similar in terms of how it is treated): apache/datafusion#11989 (comment)

Update; filed apache/datafusion#12560 to track the regression more

[rtyler]

@alamb thanks for following up! I feel a little sheepish 🐑 insofar that my internal CI showed these errors weeks ago in my integration testing between datafusion main with delta-rs main, I just didn't have the time to triage them like @ion-elgreco has 😄

I've subscribed to the issue you created and can help testing if necessary! Depending on my availability this weekend I can try fixing it in datafusion too 🤔

[matthewmturner]

Hi there - I believe a fix was implemented in datafusion and this PR is not blocked any longer.

[ion-elgreco]

Hi there - I believe a fix was implemented in datafusion and this PR is not blocked any longer.

It's not yet released though

[rtyler]

I can confirm that the latest datafusion main passes tests in CI 🥳

[alamb]

fix

I filed apache/datafusion#12813 to consider releasing a patch version of DataFusion

[alamb]

We plan to make another DataFusion patch release to unblock this upgrade:

• Release Minor DataFusion 43.1.0 release apache/datafusion#13499

[alamb]

[alamb]

👋 hey there -- we are starting to get ready for the DataFusion 44 release and are trying to make sure it is less disruptive to delta-rs than this past release

• Release DataFusion 44.0.0 apache/datafusion#13334

@rtyler Would it be ok if we started working on the upgrade now before the DataFusion release to test and find any potential breakages before?

Maybe we can make a WIP PR to this repo with a crates.io pin (similar to what i am doing in DataFusion with arrow-rs):

• WIP Upgrade to arrow-rs/parquet 54.0.0 apache/datafusion#13663

If this sounds ok, I can start working on that PR

[roeap]

hey @alamb - that would not only be OK, but a great thing to do :). Happy to support if needed along the way.

[alamb]

Cool -- thanks -- I'll give it a try hopefully in the next day or two. I am sure I will need help with the python stuff

[alamb]

I haven't made any actual progress on this, but I have created a ticket to track the work:

• Test DataFusion 44.0.0 with delta.rs apache/datafusion#13834

[rtyler]

@alamb I had not seen this issue last week, sorry! I maintain a branch for customers with datafusion main on it, I will cherry-pick and queue up the delta-rs modifications I have shortly

[alamb]

@alamb I had not seen this issue last week, sorry! I maintain a branch for customers with datafusion main on it, I will cherry-pick and queue up the delta-rs modifications I have shortly

No worries at all -- thank you 🙏

I was actually about to start work on trying to upgrade DataFuson in delta.rs this morning (I have a few other things to do first) but if you have already done the work that is amazing. Thank you!!!!

[rtyler]

#3073

[alamb]

#3073

😍