wip feat(ingest/databricks): ingest lineage between Delta paths

[ligfx]

No description provided.

[codecov]

❌ 5 Tests Failed:

Tests completed	Failed	Passed	Skipped
4358	5	4353	65

View the top 3 failed test(s) by shortest run time

tests.lineage.test_lineage_sdk::test_column_level_lineage_from_schema_field

Stack Traces | 0.037s run time

test_client = <datahub.sdk.main_client.DataHubClient object at 0x7fa88adc2110>
test_datasets = {'downstream1': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_downstream_001,PROD)'), 'downstrea...ream_003,PROD)'), 'upstream': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_upstream_001,PROD)')}

    def test_column_level_lineage_from_schema_field(
        test_client: DataHubClient, test_datasets: Dict[str, Dataset]
    ):
        source_schema_field = SchemaFieldUrn(test_datasets["upstream"].urn, "id")
        column_lineage_results = test_client.lineage.get_lineage(
            source_urn=str(source_schema_field), direction="downstream", max_hops=3
        )
    
>       assert len(column_lineage_results) == 3
E       assert 0 == 3
E        +  where 0 = len([])

tests/lineage/test_lineage_sdk.py:203: AssertionError

tests.lineage.test_lineage_sdk::test_filtered_column_level_lineage

Stack Traces | 0.064s run time

test_client = <datahub.sdk.main_client.DataHubClient object at 0x7fa88adc2110>
test_datasets = {'downstream1': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_downstream_001,PROD)'), 'downstrea...ream_003,PROD)'), 'upstream': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_upstream_001,PROD)')}

    def test_filtered_column_level_lineage(
        test_client: DataHubClient, test_datasets: Dict[str, Dataset]
    ):
        filtered_column_lineage_results = test_client.lineage.get_lineage(
            source_urn=str(test_datasets["upstream"].urn),
            source_column="id",
            direction="downstream",
            max_hops=3,
            filter=F.and_(F.platform("mysql"), F.entity_type("dataset")),
        )
    
>       assert len(filtered_column_lineage_results) == 1
E       assert 0 == 1
E        +  where 0 = len([])

tests/lineage/test_lineage_sdk.py:185: AssertionError

tests.lineage.test_lineage_sdk::test_table_level_lineage

Stack Traces | 0.162s run time

test_client = <datahub.sdk.main_client.DataHubClient object at 0x7fa88adc2110>
test_datasets = {'downstream1': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_downstream_001,PROD)'), 'downstrea...ream_003,PROD)'), 'upstream': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_upstream_001,PROD)')}

    def test_table_level_lineage(
        test_client: DataHubClient, test_datasets: Dict[str, Dataset]
    ):
        table_lineage_results = test_client.lineage.get_lineage(
            source_urn=str(test_datasets["upstream"].urn),
            direction="downstream",
            max_hops=3,
        )
    
>       assert len(table_lineage_results) == 3
E       assert 0 == 3
E        +  where 0 = len([])

tests/lineage/test_lineage_sdk.py:109: AssertionError

tests.integration.unity.test_unity_catalog_ingest::test_ingestion

Stack Traces | 1.75s run time

Metadata files differ (use `pytest --update-golden-files` to update):
Urn changed, urn:li:dataset:(urn:li:dataPlatform:databricks,hive_metastore.bronze_kambi.external_metastore,PROD):
<upstreamLineage> removed

Urn changed, urn:li:dataset:(urn:li:dataPlatform:databricks,quickstart_catalog.quickstart_schema.quickstart_table_external,PROD):
<upstreamLineage> removed

tests.lineage.test_lineage_sdk::test_column_level_lineage

Stack Traces | 10.5s run time

test_client = <datahub.sdk.main_client.DataHubClient object at 0x7fa88adc2110>
test_datasets = {'downstream1': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_downstream_001,PROD)'), 'downstrea...ream_003,PROD)'), 'upstream': Dataset('urn:li:dataset:(urn:li:dataPlatform:snowflake,test_lineage_upstream_001,PROD)')}

    def test_column_level_lineage(
        test_client: DataHubClient, test_datasets: Dict[str, Dataset]
    ):
        column_lineage_results = test_client.lineage.get_lineage(
            source_urn=str(test_datasets["upstream"].urn),
            source_column="id",
            direction="downstream",
            max_hops=3,
        )
    
>       assert len(column_lineage_results) == 3
E       assert 0 == 3
E        +  where 0 = len([])

tests/lineage/test_lineage_sdk.py:152: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[alwaysmeticulous]

🔴 Meticulous spotted visual differences in 20 of 1435 screens tested: view and approve differences detected.

Meticulous evaluated ~8 hours of user flows against your PR.

_{Last updated for commit d0b6a0b. This comment will update as new commits are pushed.}