Configurable Error Handling for Invalid Timestamp Values During Loads

[vai-airbyte]

Problem
Syncs can fail when a record contains a malformed timestamp value (for example, an invalid year format). Currently, a single invalid value can cause the entire sync to fail, even if all other records are valid.

These malformed values often originate from source systems due to data entry errors or legacy data and may not be easily correctable upstream.

Example Error (Sanitized)

Warning from destination: Timestamp '+12018-02-28T00:00' is not recognized
File 'destination_file.csv.gz', line XXXX, character XXX
Row XXXX, column "example_table"["example_timestamp_column"]
If you would like to continue loading when an error is encountered,
use values such as 'SKIP_FILE' or 'CONTINUE' for the ON_ERROR option.

Proposed Solution
Provide a configurable option in Airbyte that allows users to control how destination load errors are handled. Possible behaviors could include:

• Skipping invalid records
• Continuing the load while logging errors
• Failing the sync (current behavior)

This would allow valid data to be ingested successfully even when isolated data quality issues exist.

Alternatives Considered
Fixing the data at the source system was considered, but this is not always practical due to access limitations, long remediation cycles, or external ownership of the source. Handling errors within Airbyte provides a more flexible and scalable solution.

Additional Context
Many destinations (including Snowflake) already support configurable load behavior when encountering invalid data. Exposing these controls in Airbyte would better align with destination capabilities and improve robustness for real-world datasets.