Questions for serialization

[gaoyuanning]

Hi, I have some concerns for geometry serialization. From my understanding, both duckdb_spatial and postgis take geometry type, point count, and point array for serialization. In addition to that, postgis also serializes the information of GBOX (bouding box), srid, flags. So my concerns are as follows:

1. From my understanding, one possible application for GBOX is to quickly filter geometries, e.g., when used for R-tree index. So in later development, will duckdb_spatial dynamically calculate the bounding box, e.g., when executing corresponding quries or constructing R-tree index?
2. For SRID (spatial reference identifier), will duckdb_spatial use other metadata columns to store SRID information for geometry columns? Otherwise, how do we distinguish coordinated systems for different geometry columns?
3. postgis uses flags to store Z, M flag, which denotes a 3-D or 4-D geometry. So if duckdb_spatial needs to support 3-D geometry in later develoment, should we consider such serialization information?
4. One minor issue: duckdb_spatial uses 4 bytes to store the Type. Is uint_8 enough to store such information?

// duckdb_spatial
// Data (variable length)
// -- Point
//     Type ( 4 bytes)
//    Count (4 bytes) (count == 0 if empty point, otherwise 1)
//    X (8 bytes)
//    Y (8 bytes)
// -- LineString
//    Type (4 bytes)
//    Length (4 bytes)
//    Points (variable length)

// postgis
liblwgeom/liblwgeom.h.in

typedef struct
{
    GBOX *bbox;
    void *data;
    int32_t srid;
    lwflags_t flags;
    uint8_t type;
    char pad[1]; /* Padding to 24 bytes (unused) */
}
LWGEOM;

typedef struct
{
    lwflags_t flags;
    double xmin;
    double xmax;
    double ymin;
    double ymax;
    double zmin;
    double zmax;
    double mmin;
    double mmax;
} GBOX;

[Maxxen]

Thanks again for the questions! I love to see this sort of discussion being created.

1. Maybe, I think this depends on the technical details on how we implement persistent spatial indexes, AFAIK the bbox would have to be stored separately in the index as well (not sure how this works in postgres), but having the bbox cached in the geometry itself is probably useful when doing spatial predicates even without an index.

2. I'm torn on where to store SRID, my experience is that you pretty much never want to have a column with mixed SRID's, so storing them "inside" the geometry doesn't really make sense to me. GeoPandas only allows a single SRID per geometry column, but in the case of DuckDB I think it would be better to store them in a separate column as a string/enum and benefit from DuckDB's dictionary compression, which is why the ST_Transform functions takes source and destination SRID as parameters, and have optimizations for when the params are constant. On the other hand storing the SRID in the geometry is part of the standard, and is part of the geometry model ogr/gdal/postgis expects. Until we have a proper SPATIAL_REF_SYS table it doesn't really matter though.

3. We already store a uint8_t of flags as part of theGeometryPrefix struct which is serialized, the intention is to store dimension information and other metadata there in the future.

4. We use uint32_t to store the type information in order to make sure that the subsequent vertex data is aligned to 8 bytes. This is the same representation used in lwgeom which makes memory access faster at the cost of some storage. Although I have been told that this is not as important on modern processors, I don't have any benchmarks on it.

[gaoyuanning]

Thanks very much for the detailed reply! For answer#2, I also agree that all of the gemotries in a column have the same SRID.