Unable to perform WHERE queries on partitioned column

[Johanneshn]

I have a dataset partitioned on a specific column using delta-rs. I encounter an exception when I execute a SELECT query with a WHERE clause targeting the partition column.

Query:
select * from delta_scan('D://partitioned') WHERE PartitionColumn= 1 LIMIT 10;

Error:
IO Error: Hit DeltaKernel FFI error (from: While trying to read from delta table: 'D://partitioned/'): Hit error: 2 (ArrowError) with message (Json error: whilst decoding field 'minValues': Encountered unmasked nulls in non-nullable StructArray child: Field { name: "PartitionColumn", data_type: Int32, nullable: false, dict_id: 0, dict_is_ordered: false, metadata: {} })

Other queries on this table work (and WHERE clause targeting other columns)

[jorritsandbrink]

I experience the same. Maybe this has to do with the fact that the partition column does not exist in the underlying Parquet files and must be read from the transaction log instead.

[rudolfix]

I was able to modify the metadata of delta tables produced by delta-rs so duckdb can filter partition now. Root cause is stats field in the metadata where it contains (serialized) maxValues and minValues which are rows id parquet files in partition. and those rows are missing partition columns (like the data itself). when I add this manually, duckdb is able to select data from it. it also skips filtered out partitions 🚀 🤯

I'm not sure who's wrong here: delta kernel or delta rs. @samansmink what is your experience with delta-kernel? are bugs fixed quickly? otherwise @jorritsandbrink maybe we should file this in delta-rs. they were AFAIK pretty responsive

[thomas-chauvet]

delta-rs team is super responsive, you can open a PR there, they will discuss if it should move to delta-kernel

I've personnally added partition column into the data directly but I still have a weird issue when filtering on it (despite also being in the data)