Getting Original Schema of a DataFile in a FileScanTask?

[srilman]

Question

Is there a recommended way to getting the base / original schema or schema-id of a data file in a FileScanTask returned during FileTableScan.plan_files? This is useful to determine what kind of schema evolution occurred with the subset of files we are reading, and group files together with the same schemas for reads.

I had a hard time accomplishing this in the Java library, but found it much easier to do in Python. In plan_files, we can get the snapshot id a data file was created by looking at the snapshot_id of the associated manifest entry (or added_snapshot_id of the manifest list if the previous is null). From there, we can get the associated schema per snapshot.

Is this a logical approach, or is there a better way to get the original schema? Happy to open a PR to integrate this into FileTableScan if it would be useful!

[srilman]

@Fokko any thoughts?

[Fokko]

Hey @srilman sorry for not replying earlier. It slipped somewhere through the cracks, thanks for pinging me!

Is there a recommended way to getting the base / original schema or schema-id of a data file in a FileScanTask returned during FileTableScan.plan_files? This is useful to determine what kind of schema evolution occurred with the subset of files we are reading, and group files together with the same schemas for reads.

Unfortunately, no :(

From there, we can get the associated schema per snapshot.

A snapshot can have files with multiple schemas. See the example in duckdb/duckdb-iceberg#40

Is this a logical approach, or is there a better way to get the original schema? Happy to open a PR to integrate this into FileTableScan if it would be useful!

The only source of truth is in the Parquet file. The crux is that you don't want to do lookups based on names, but only on field field-IDs. Names can change over time, but field-id won't. So if you filter on a specific field-id, that will always work correctly. The same goes when reading parquet files, it can be that the names change over time, and you don't want to rewrite petabytes of data, so you can leave them as is. When reading, you want to make the field-ids to the current schema.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'