Skip to content

Specific schema evolution behavior with Pydantic "extra" setting #3537

@serl

Description

@serl

dlt version

1.20.0

Describe the problem

I'm currently using dlt with Pydantic for data validation and schema definition, and I'm encountering an issue with schema evolution and Pydantic's extra setting.

Here's my desired behavior and the current constraint:

  1. I want to maintain a strict schema validation, not allowing any extra columns during validation that are not defined in my Pydantic model. To achieve this, I'm setting Pydantic's extra=forbid.

  2. At the same time, I want my destination table to evolve (i.e., add new columns) as I update my Pydantic model schema. For that, I set dlt's schema_contract={"columns": "evolve"}.

The issue I'm facing is that even with these settings, dlt adds extra columns that are not defined in my Pydantic model, behaving as if Pydantic's extra=allow. This seems to imply that dlt is not respecting the extra=forbid setting in Pydantic when adding new columns to the table.

I believe dlt should give an option to respect Pydantic's extra settings while managing schema evolution. As of now, I'm having to change Pydantic extra settings back and forth when adding columns to existing tables, which is error-prone.

Expected behavior

No response

Steps to reproduce

Here's an approximate code outline for reference:

class MyModel(BaseModel):
    class Config:
        extra = "forbid"

    id: int
    # ...

@dlt.resource(name="my_resource", columns=MyModel, schema_contract={"columns": "evolve"})
def get_my_resource():
    ...

Operating system

macOS

Runtime environment

Local

Python version

3.13

dlt data source

Rest API

dlt destination

Snowflake

Other deployment details

No response

Additional information

I'd be happy to help with a PR

Metadata

Metadata

Assignees

Labels

questionFurther information is requested

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions