Initial PMTiles support

[tdcosta100]

This PR introduces the PMTiles support, which was first asked in #1978. It's experimental for now, and needs more testing, so I need everyone interested in this feature to test it, so we can mitigate problems that can arise from my modifications.

[github-actions]

Bloaty Results 🐋

Compared to main

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.1% +1.65Mi  +1.2%  +379Ki    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-2882-compared-to-main.txt

Compared to d387090 (legacy)

    FILE SIZE        VM SIZE    
 --------------  -------------- 
   +31% +36.2Mi  +439% +26.2Mi    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results/pr-2882-compared-to-legacy.txt

[github-actions]

Benchmark Results ⚡

Benchmark                                                     Time             CPU      Time Old      Time New       CPU Old       CPU New
------------------------------------------------------------------------------------------------------------------------------------------
OVERALL_GEOMEAN                                            +0.0040         +0.0037             0             0             0             0

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/benchmark-results/pr-2882-compared-to-main.txt

[github-actions]

Bloaty Results (iOS) 🐋

Compared to main

    FILE SIZE        VM SIZE    
 --------------  -------------- 
  +1.6%  +210Ki  +1.5%  +192Ki    TOTAL

Full report: https://maplibre-native.s3.eu-central-1.amazonaws.com/bloaty-results-ios/pr-2882-compared-to-main.txt

[JesseCrocker]

I could probably figure this out from reading the code, but does this add support for reading local pmtiles files, reading over http, or both?

[louwers]

Great stuff!

Needs some tests. Due to the binary size increase we also need to make sure there is an opt-out with a preprocessor macro

[tdcosta100]

I could probably figure this out from reading the code, but does this add support for reading local pmtiles files, reading over http, or both?

It is agnostic about it, it just makes requests to the Resource Loader, then it use the appropriate source, that is, you can use http://, https:// or file://, which one is better for your scenario.

[tdcosta100]

Great stuff!

Needs some tests. Due to the binary size increase we also need to make sure there is an opt-out with a preprocessor macro

Thank you! It was a great effort to achieve this implementation, but I'm still learning how to make such a component integration properly. Some adjustements will be made. Since the pmtiles_file_source.cpp is added in the Bazel/CMake files, maybe I can add a new switch in CMake to opt-out the PMTiles support, something like MLN_WITH_PMTILES=OFF if you don't want it to be included by default.

[tdcosta100]

Great stuff!

Needs some tests. Due to the binary size increase we also need to make sure there is an opt-out with a preprocessor macro

Hi, @louwers. Could you check also the last commit, please? I'm not sure if I took the best way to control the PMTiles code using CMake.

[louwers]

I think all upper case names should be reserved for macros. https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines#nl9-use-all_caps-for-macro-names-only

[tdcosta100]

Originally I wanted to declare these numbers as defines, but how to use them correctly? The compiler complains when I define them using #define.

[louwers]

Keep it constexpr, but lowercase them.

Also camelcase maybe pmtilesHeaderOffset?

[louwers]

Looks OK!

I would prefer as little #defines in the codebase as possible. Maybe zero, with a dummy implementation available?

Some tests are also still needed.

[tdcosta100]

Looks OK!

I would prefer as little #defines in the codebase as possible. Maybe zero, with a dummy implementation available?

Some tests are also still needed.

Interesting. My implementation was in direction of vanishing alway everything if MLN_WITH_PMTILES=OFF. What a dummy implementation would do? Implement the required methods as stubs doing nothing? Display an error telling PMTiles support is not available? Because if so, I'm afraid the binary footprint reduction will be no that great. Anyway, I will adjust to either scenario you will tell me to do. I'm open to suggestions.

[louwers]

I want to avoid #ifdefs sprinkled across the codebase, because it makes code hard to read and reason about.

The dummy implementation would be another file that does not pull in the PMTiles library as a dependency, which is where the bulk of the binary size increase is coming from. The dummy implementation would just always return false from canRequest().

[tdcosta100]

Okay, I will think about it and implement these changes. Without #ifdefs everywhere, of course hahaha.

[tdcosta100]

We need C++ unit tests and ideally also some tests for Android and iOS.

Let me know if I can help.

Theoretically every OS will include the new PMTiles tests, but now I need to fix the docker image of the Qt5 tests. Maybe @ntadej could help us making a new image with CMake 3.24.

I think we can drop Qt5 support at this point. I'm travelling this week, but I can update the tests once I'm back if this is not too late.

Hi, @ntadej! Could you please update the Qt5 Docket image to have CMake 3.24? I moved the new tests to #2968.

[louwers]

@tdcosta100 Please re-request a review when you want me to take a another look. Thanks :-)

[tdcosta100]

@tdcosta100 Please re-request a review when you want me to take a another look. Thanks :-)

I'd like another review, thanks :). Also, I need to run the tests with the new Windows-CI workflow, but I need @ntadej help with Qt workflows.

[louwers]

I tried it out with wipfli's FourSquare POI PMTiles.

https://wipfli.github.io/foursquare-os-places-pmtiles/style.json

The protomaps PMTiles source is not loading on iOS. The POIs do load and are visible.

On Android nothing is visible when using this style

Seems to be some font loading issue. When I remove all Symbol layers the layers using the protomapssource also load but I get interesting effects (on macOS).

https://gist.githubusercontent.com/louwers/97ef201ce10f06392662addd4704be12/raw/fc3d78446f3fc9e9d1aaebb07464e205112980ca/u3gveoe.json

bazel run //platform/glfw:glfw_app --//:renderer=metal -- --style https://gist.githubusercontent.com/louwers/97ef201ce10f06392662addd4704be12/raw/fc3d78446f3fc9e9d1aaebb07464e205112980ca/u3gveoe.json

Can't seem to reproduce it second time around.

Android:

2024-11-22 15:16:57.088 12254-12254 Mbgl org.maplibre.android.testapp E {android.testapp}[Style]: Failed to load source foursquare-10M: Error fetching PMTiles header: HTTP status code 206

[louwers]

Now seems fully functional on Android!

Untitled.mp4

[tdcosta100]

@louwers That's nice! So now the only requirement is the windows-ci branch to be merged and then after all tests we can finally merge this. Or we can merge it anyway.

[louwers]

The fact that the Android tests were passing, while in fact PMTiles did not work means we need additional test coverage.

Edit: Oh the network tests do not run on Android. That is probably why.

[tdcosta100]

The fact that the Android tests were passing, while in fact PMTiles did not work means we need additional test coverage.

Edit: Oh the network tests do not run on Android. That is probably why.

I agree, probably this is the cause, but I still need help to verify if coverage tests are enough for PMTiles.