Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Support geometry data type for Unit Tests #142

Open
2 tasks done
magno32 opened this issue Aug 20, 2024 · 1 comment
Open
2 tasks done

[FEATURE] Support geometry data type for Unit Tests #142

magno32 opened this issue Aug 20, 2024 · 1 comment
Labels
enhancement New feature or request unit tests Issues related to built-in dbt unit testing functionality

Comments

@magno32
Copy link

magno32 commented Aug 20, 2024

Is this a new bug?

  • I believe this is a new bug
  • I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Probably related to #13; Generated SQL for unit tests creates a cast of geometry columns (probably any custom type) to USER-DEFINED, which is invalid.

with __dbt__cte__int_county_state__joined as (

-- Fixture for int_county_state__joined
    select cast(null as integer)              as county_id,
           cast(null as integer)              as state_id,
           cast(null as integer)              as region_id,
           cast(null as integer)              as division_id,
           cast(null as character varying)    as name,
           cast(null as character varying)    as name_county,
           cast(null as character varying)    as state_name,
           cast(null as character varying(2)) as state_abbreviation,
           cast(null as character varying)    as region_name,
           cast(null as character varying)    as division_name,
           cast(null as bigint)               as area_land,
           cast(null as bigint)               as area_water,
           cast(null as USER - DEFINED)       as center,

           cast('POINT(0,0)' as USER - DEFINED)
                                              as geometry),
     __dbt__cte__stg_us_census__county_geo_genz as (

-- Fixture for stg_us_census__county_geo_genz
         select cast(null as integer) as county_id,

                cast('POINT(0,0)' as USER - DEFINED)
                                      as geometry)
SELECT c1.*,
       ST_asText(c1.center)   AS center_wkt,
       c2.geometry            AS geometry_simplified,
       ST_asText(c1.geometry) AS geometry_wkt,
       ST_asText(c2.geometry) AS geometry_simplified_wkt
FROM __dbt__cte__int_county_state__joined c1
         LEFT JOIN __dbt__cte__stg_us_census__county_geo_genz c2
                   ON c1.county_id = c2.county_id

This affects testing both geometry columns and standard typed columns when the base tables contain geometry.

If you apply the macro suggested in #13 (comment), you get past the incorrect cast, but the mock geometry is wrapped as though its a string.

   Database Error
    parse error - invalid geometry
    LINE 20:     cast('POINT(0,0)' as geometry)

Another note if it is relevant to a fix, this also is present when using a basic test from dbt_expectations, hopefully it uses the same rendering mechanism.

data_tests:
     - dbt_expectations.expect_column_values_to_be_of_type:
            column_type: geometry

Expected Behavior

I was hesitant to raise this issue on this adapter, since it's a postgis issue, but it feels like there should be away to configure the adapter to be aware of additional data-types.

Unit-tests should be possible when non-standard columns are present.

Steps To Reproduce

  1. Create a unit test for a table with inputs:
unit_tests:
  - name: test_county_data_integrity
    model: county
    given:
      - input: ref('int_county_state__joined')
        rows:
          - geometry: POINT(0,0)
      - input: ref('stg_us_census__county_geo_genz')
        rows:
          - geometry: POINT(0,0)
    expect:
      rows:
        - geometry: POINT(0,0)
  1. Run dbt test
  2. Tests will fail with invalid sql

Relevant log output

An error occurred during execution of unit test 'test_county_data_integrity'. There may be an error in the unit test definition: check the data types.
   Database Error
    syntax error at or near "USER"
    LINE 18: ... cast(null as bigint) as area_water, cast(null as USER-DEFIN...

Environment

- OS: mint
- Python:  3.10.12
- dbt-postgres:  1.8.2

Additional Context

No response

@magno32 magno32 added bug Something isn't working triage labels Aug 20, 2024
@JeyP91
Copy link

JeyP91 commented Aug 26, 2024

Hey, we are facing the same issue with our unit tests and custom enum types.

We have types created like this:
CREATE TYPE custom_type AS ENUM ('a', 'b', 'c');

Any tables, that are using this type cannot be mocked away, because the generated SQL is invalid.
Database Error syntax error at or near "USER".

If you need more information on our use case, I can also add more details and examples.
Thanks!

@amychen1776 amychen1776 added unit tests Issues related to built-in dbt unit testing functionality and removed triage labels Aug 28, 2024
@amychen1776 amychen1776 changed the title [Bug] Unit-Tests generate invalid sql with geometry data type [FEATURE] Support geometry data type for Unit Tests Aug 28, 2024
@amychen1776 amychen1776 added enhancement New feature or request and removed bug Something isn't working labels Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request unit tests Issues related to built-in dbt unit testing functionality
Projects
None yet
Development

No branches or pull requests

3 participants