fix: use projected_table_schema for projection in DeltaSchemaAdapter

[jkylling]

Description

After upgrading from deltalake 0.20.1 to 0.22.3 it looks like Parquet column projection is broken when using DeltaTable::scan. Instead of scanning only the a single column, it looks like all columns are fetched from storage.

Inspection with a debugger reveals that the adapted_projections are wrong here:
https://github.com/apache/datafusion/blob/88f58bf929167c5c5e2250ad87caa88d4dff11e5/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L153-L159 The adapted_projections are obtained in

delta-rs/crates/core/src/delta_datafusion/schema_adapter.rs

Lines 46 to 60 in 5b2f46b

	let mut projection = Vec::with_capacity(file_schema.fields().len());
	for (file_idx, file_field) in file_schema.fields.iter().enumerate() {
	if self.table_schema.fields().find(file_field.name()).is_some() {
	projection.push(file_idx);
	}
	}
	Ok((
	Arc::new(SchemaMapping {
	projected_schema: self.projected_table_schema.clone(),
	table_schema: self.table_schema.clone(),
	}),
	projection,
	))

Changing line 49 to use the projected_table_schema seems to solve the problem.

Opening this to see if we have test coverage.

[github-actions]

ACTION NEEDED

delta-rs follows the Conventional Commits specification for release automation.

The PR title and description are used as the merge commit message. Please update your PR title and description to match the specification.

[jkylling]

For reference, the corresponding code in Datafusion 42 is https://github.com/apache/datafusion/blob/a43ce8bd67e8d649e0e4f5260f8a5e6e10d62dbc/datafusion/core/src/datasource/physical_plan/parquet/opener.rs#L80-L82 When upgrading from 42 to 43 the meaning of table_schema was changed from being the projected schema, to be the full table schema, and a new argument projected_schema was added. This was mapped to projected_table_schema and table_schema in our code, but we forgot to update all uses of table_schema in the DeltaSchemaAdapter to use the projected_table_schema.

The changes to the SchemaAdapter API were introduced in apache/datafusion#12135

[rtyler]

I will have this merged one main goes green again

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 72.42%. Comparing base (b241bf6) to head (7a89c3b).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3068      +/-   ##
==========================================
+ Coverage   72.40%   72.42%   +0.02%     
==========================================
  Files         128      128              
  Lines       41007    41012       +5     
  Branches    41007    41012       +5     
==========================================
+ Hits        29690    29703      +13     
+ Misses       9425     9424       -1     
+ Partials     1892     1885       -7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.