-
Notifications
You must be signed in to change notification settings - Fork 430
Description
dlt version
1.20.0
Describe the problem
I'm currently using dlt with Pydantic for data validation and schema definition, and I'm encountering an issue with schema evolution and Pydantic's extra setting.
Here's my desired behavior and the current constraint:
-
I want to maintain a strict schema validation, not allowing any extra columns during validation that are not defined in my Pydantic model. To achieve this, I'm setting Pydantic's
extra=forbid. -
At the same time, I want my destination table to evolve (i.e., add new columns) as I update my Pydantic model schema. For that, I set dlt's
schema_contract={"columns": "evolve"}.
The issue I'm facing is that even with these settings, dlt adds extra columns that are not defined in my Pydantic model, behaving as if Pydantic's extra=allow. This seems to imply that dlt is not respecting the extra=forbid setting in Pydantic when adding new columns to the table.
I believe dlt should give an option to respect Pydantic's extra settings while managing schema evolution. As of now, I'm having to change Pydantic extra settings back and forth when adding columns to existing tables, which is error-prone.
Expected behavior
No response
Steps to reproduce
Here's an approximate code outline for reference:
class MyModel(BaseModel):
class Config:
extra = "forbid"
id: int
# ...
@dlt.resource(name="my_resource", columns=MyModel, schema_contract={"columns": "evolve"})
def get_my_resource():
...Operating system
macOS
Runtime environment
Local
Python version
3.13
dlt data source
Rest API
dlt destination
Snowflake
Other deployment details
No response
Additional information
I'd be happy to help with a PR
Metadata
Metadata
Assignees
Labels
Type
Projects
Status