[Feature] geometry and geography logical type implementations

[kaushiksrini]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Describe the solution you'd like

• Support for Geometry/Geography type for Parquet, as it was added to thrift definition (PARQUET-2471: Add GEOMETRY and GEOGRAPHY logical types parquet-format#240) and will be included arrow 21.0.0 with GH-45522: [Parquet][C++] Parquet GEOMETRY and GEOGRAPHY logical type implementations arrow#45459 change.
• Add native support for the parquet crate to handle these new types to align with other ecosystems.

Describe alternatives you've considered

• Alternatives are being implemented in PyArrow and Arrow-cpp.

Additional context

• Happy to work on this feature.

[alamb]

Thank you @kaushiksrini -- this sounds like a great addition to me

FYI @kylebarron and @paleolimbot -- I believe you may have some interest in this topic. I am no 100% sure how this is related to geo-arrow / geo-parquet but perhaps it is

Also FYI @timsaucer who I think also may be interested

[alamb]

In terms of implementation, I recommend we follow a similar approach to the variant implementation, specifically:

1. The use of geometry types should be optional (not required when using parquet-rs)
2. The code should live in a crate similar to parquet-geometry or parquet-geo
3. The arrow implementation should use the arrow extension type mechanism (not add a type to DataType, for example)

One specific approach might be to follow @PinkCrow007 's example in the following PR -- that is to sketch out what this support might look like in a PR which we then break up into smaller parts to keep it going.

• Variant Support for Arrow and Parquet [DRAFT] #7404

[paleolimbot]

Thanks for sketching that out! I think that sounds like a reasonable plan...I'm completely new to how things are structured in arrow-rs. I had been planning to take this on but had to ensure I completed a bunch of other things I said I would do and am happy to review if anybody is interested in getting there first.

The first things I need are the logical type definition and the statistics from the ParquetMetadata, although full support for geoarrow.wkb in and out + write statistics are also great.

[kaushiksrini]

Thanks @alamb and @paleolimbot for taking a look at this issue! I’d be happy to have a stab at implementing this—I'll put together a draft and break it into smaller chunks to make the review easier.

[kylebarron]

Thanks for creating the issue @kaushiksrini!

I am no 100% sure how this is related to geo-arrow / geo-parquet but perhaps it is

GeoParquet is a community-driven approach/spec to storing vector geospatial data in Parquet. It uses key-value metadata to describe what column(s) are geospatial, but doesn't modify the format at all; all GeoParquet files are still normal Parquet files. Read/write support for GeoParquet can (and should) be implemented totally external to the parquet crate, e.g. in my geoparquet crate.

Then after some time the Parquet community decided they were open to creating a native Parquet logical type for geospatial data. This native logical type is the "next version" of GeoParquet so to speak. This logical type actually modifies some structures in the Parquet file, and is what this issue is about, if I understand correctly.

GeoArrow is an extension to Arrow to convey logical type information on top of arrow arrays. Ideally the Parquet logical type would read/write some sort of GeoArrow data. Most simply, it would read the binary buffers from Parquet and then just add the geoarrow.wkb Arrow extension name.

I’d be happy to have a stab at implementing this—I'll put together a draft and break it into smaller chunks to make the review easier.

Great! I'd be happy to review a draft too!

[paleolimbot]

Most simply, it would read the binary buffers from Parquet and then just add the geoarrow.wkb Arrow extension name.

...and convert the CRS! The logic in https://github.com/apache/arrow/blob/c8fe26898ce49c58514f511be58afddce176826b/cpp/src/parquet/geospatial/util_json_internal.cc is self-contained and should work in Rust, too.

[sdf-jkl]

Does this issue address the same problem as #7240?

[paleolimbot]

As far as I know, yes!

[sdf-jkl]

@kaushiksrini If you need any help with this issue, I’d love to contribute. Just let me know how I can assist!

[kaushiksrini]

Hi @sdf-jkl, started working on this with inspiration from the variant draft PR linked above. I should have a draft ready for review soon and would appreciate your help in pushing it forward then!

[BlakeOrth]

Subscribed, as I'm also interested in having this functionality in the core parquet implementation and am willing to contribute.

[kylebarron]

I started a very rough initial implementation in #8222