Fix node location OOM error and hang (#1464)

[msbarry]

Fix a race deadlock condition that can happen when --nodemap-type=array --storage=mmap are used together and an exception gets thrown during pass1 when processing nodes. Also fixed the specific exception people were seeing in #1464 by allocating smalller chunks of memory.

Fixes #1464

[github-actions]

This Branch 1bb40f9

Base 8c0a4a6

0:01:10 DEB [archive] - Tile stats: 0:01:10 DEB [archive] - Biggest tiles (gzipped) 1. 14/4942/6092 (161k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:88k) 2. 9/154/190 (148k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k) 3. 10/308/381 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k) 4. 10/308/380 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k) 5. 14/4941/6092 (117k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:69k) 6. 14/4941/6093 (117k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k) 7. 14/4946/6113 (110k) https://onthegomap.github.io/planetiler-demo/#14.5/41.48389/-71.31226 (building:59k) 8. 14/4946/6112 (109k) https://onthegomap.github.io/planetiler-demo/#14.5/41.50035/-71.31226 (building:67k) 9. 14/4940/6092 (101k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k) 10. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k) 0:01:10 DEB [archive] - Max tile sizes z0 z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 z12 z13 z14 all boundary 151 336 409 544 802 287 382 490 670 1.6k 2k 6.8k 6.2k 5.6k 4.4k 6.8k water 7.7k 3.7k 8.6k 5.5k 2.6k 5.1k 15k 18k 16k 26k 15k 13k 17k 15k 12k 26k place 0 0 487 487 487 733 822 1.1k 1.7k 3.3k 6.1k 3.8k 1.9k 935 1k 6.1k landuse 0 0 0 0 549 695 1.6k 6.8k 17k 44k 58k 49k 38k 19k 12k 58k transportation 0 0 0 0 406 1k 1.4k 4.5k 6.3k 21k 15k 17k 66k 45k 35k 66k waterway 0 0 0 0 112 119 0 0 0 3.3k 2.4k 2k 2.1k 4.9k 2.4k 4.9k park 0 0 0 0 0 0 1.1k 4k 9.7k 18k 13k 8.2k 3.7k 3.4k 4.4k 18k transportation_name 0 0 0 0 0 0 291 360 1.1k 1.9k 5.5k 4.8k 3.9k 3.5k 19k 19k landcover 0 0 0 0 0 0 0 9.7k 29k 85k 72k 81k 53k 30k 25k 85k mountain_peak 0 0 0 0 0 0 0 1.1k 1.8k 3.4k 4.4k 2.8k 1.4k 1.4k 869 4.4k water_name 0 0 0 0 0 0 0 0 0 486 461 433 452 1.2k 1.5k 1.5k aerodrome_label 0 0 0 0 0 0 0 0 0 0 666 289 273 221 221 666 aeroway 0 0 0 0 0 0 0 0 0 0 1.6k 2k 3k 3.3k 2.8k 3.3k poi 0 0 0 0 0 0 0 0 0 0 0 0 589 586 88k 88k building 0 0 0 0 0 0 0 0 0 0 0 0 0 59k 92k 92k housenumber 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35k 35k full tile 7.9k 4k 9.5k 6.5k 3.7k 6.3k 21k 41k 84k 201k 184k 135k 114k 127k 253k 253k gzipped 6.2k 3.5k 7.1k 5.2k 3.1k 5k 14k 29k 60k 148k 137k 99k 84k 90k 161k 161k 0:01:10 DEB [archive] - Max tile: 253k (gzipped: 161k) 0:01:10 DEB [archive] - Avg tile: 5.5k (gzipped: 4.1k) using weighted average based on OSM traffic 0:01:10 DEB [archive] - # tiles: 4,115,031 0:01:10 DEB [archive] - # features: 5,738,475 0:01:10 INF [archive] - Finished in 21s cpu:1m15s avg:3.7 0:01:10 INF [archive] - read 1x(3% 0.6s wait:19s done:1s) 0:01:10 INF [archive] - encode 4x(57% 12s wait:2s done:1s) 0:01:10 INF [archive] - write 1x(21% 4s wait:14s done:1s) 0:01:10 INF [archive] - Finished in 1m11s cpu:3m53s gc:1s avg:3.3 0:01:10 INF [archive] - FINISHED! 0:01:10 INF [archive] - 0:01:10 INF [archive] - ---------------------------------------- 0:01:10 INF [archive] - data errors: 0:01:10 INF [archive] - render_snap_fix_input 16,942 0:01:10 INF [archive] - osm_multipolygon_missing_way 396 0:01:10 INF [archive] - osm_boundary_missing_way 55 0:01:10 INF [archive] - merge_snap_fix_input 15 0:01:10 INF [archive] - feature_polygon_osm_invalid_multipolygon_empty_after_fix 7 0:01:10 INF [archive] - feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix 2 0:01:10 INF [archive] - omt_fix_water_before_ne_intersect 2 0:01:10 INF [archive] - render_snap_fix_input2 1 0:01:10 INF [archive] - omt_park_area_osm_invalid_multipolygon_empty_after_fix 1 0:01:10 INF [archive] - ---------------------------------------- 0:01:10 INF [archive] - overall 1m11s cpu:3m53s gc:1s avg:3.3 0:01:10 INF [archive] - lake_centerlines 3s cpu:6s avg:2.3 0:01:10 INF [archive] - read 1x(17% 0.5s done:2s) 0:01:10 INF [archive] - process 4x(0% 0s done:2s) 0:01:10 INF [archive] - write 1x(0% 0s done:2s) 0:01:10 INF [archive] - water_polygons 15s cpu:41s avg:2.7 0:01:10 INF [archive] - read 1x(42% 6s done:7s) 0:01:10 INF [archive] - process 4x(26% 4s wait:4s done:6s) 0:01:10 INF [archive] - write 1x(3% 0.5s wait:10s done:5s) 0:01:10 INF [archive] - natural_earth 7s cpu:14s avg:2.1 0:01:10 INF [archive] - read 1x(96% 6s) 0:01:10 INF [archive] - process 4x(13% 0.9s wait:6s) 0:01:10 INF [archive] - write 1x(0% 0s wait:6s) 0:01:10 INF [archive] - osm_pass1 2s cpu:7s avg:3.3 0:01:10 INF [archive] - read 1x(2% 0s wait:2s) 0:01:10 INF [archive] - parse 4x(34% 0.7s) 0:01:10 INF [archive] - process 1x(70% 1s) 0:01:10 INF [archive] - osm_pass2 21s cpu:1m24s avg:4 0:01:10 INF [archive] - read 1x(0% 0s wait:13s done:8s) 0:01:10 INF [archive] - process 4x(75% 16s) 0:01:10 INF [archive] - write 1x(2% 0.5s wait:21s) 0:01:10 INF [archive] - ne_lakes 0s cpu:0s avg:0 0:01:10 INF [archive] - boundaries 0s cpu:0s avg:0 0:01:10 INF [archive] - agg_stop 0s cpu:0s avg:0 0:01:10 INF [archive] - sort 1s cpu:4s avg:2.7 0:01:10 INF [archive] - worker 1x(51% 0.7s) 0:01:10 INF [archive] - archive 21s cpu:1m15s avg:3.7 0:01:10 INF [archive] - read 1x(3% 0.6s wait:19s done:1s) 0:01:10 INF [archive] - encode 4x(57% 12s wait:2s done:1s) 0:01:10 INF [archive] - write 1x(21% 4s wait:14s done:1s) 0:01:10 INF [archive] - ---------------------------------------- 0:01:10 INF [archive] - archive 108MB 0:01:10 INF [archive] - features 295MB -rw-r--r-- 1 runner runner 89M Feb 28 13:47 run.jar

0:01:10 DEB [archive] - Tile stats: 0:01:10 DEB [archive] - Biggest tiles (gzipped) 1. 14/4942/6092 (161k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.40015 (poi:88k) 2. 9/154/190 (148k) https://onthegomap.github.io/planetiler-demo/#9.5/41.77078/-71.36719 (landcover:85k) 3. 10/308/381 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.63994/-71.54297 (landcover:72k) 4. 10/308/380 (137k) https://onthegomap.github.io/planetiler-demo/#10.5/41.90214/-71.54297 (landcover:66k) 5. 14/4941/6092 (117k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.42212 (poi:69k) 6. 14/4941/6093 (117k) https://onthegomap.github.io/planetiler-demo/#14.5/41.81227/-71.42212 (building:62k) 7. 14/4946/6113 (110k) https://onthegomap.github.io/planetiler-demo/#14.5/41.48389/-71.31226 (building:59k) 8. 14/4946/6112 (109k) https://onthegomap.github.io/planetiler-demo/#14.5/41.50035/-71.31226 (building:67k) 9. 14/4940/6092 (101k) https://onthegomap.github.io/planetiler-demo/#14.5/41.82864/-71.44409 (building:92k) 10. 11/616/762 (99k) https://onthegomap.github.io/planetiler-demo/#11.5/41.7057/-71.63086 (landcover:71k) 0:01:10 DEB [archive] - Max tile sizes z0 z1 z2 z3 z4 z5 z6 z7 z8 z9 z10 z11 z12 z13 z14 all boundary 151 336 409 544 802 287 382 490 670 1.6k 2k 6.8k 6.2k 5.6k 4.4k 6.8k water 7.7k 3.7k 8.6k 5.5k 2.6k 5.1k 15k 18k 16k 26k 15k 13k 17k 15k 12k 26k place 0 0 487 487 487 733 822 1.1k 1.7k 3.3k 6.1k 3.8k 1.9k 935 1k 6.1k landuse 0 0 0 0 549 695 1.6k 6.8k 17k 44k 58k 49k 38k 19k 12k 58k transportation 0 0 0 0 406 1k 1.4k 4.5k 6.3k 21k 15k 17k 66k 45k 35k 66k waterway 0 0 0 0 112 119 0 0 0 3.3k 2.4k 2k 2.1k 4.9k 2.4k 4.9k park 0 0 0 0 0 0 1.1k 4k 9.7k 18k 13k 8.2k 3.7k 3.4k 4.4k 18k transportation_name 0 0 0 0 0 0 291 360 1.1k 1.9k 5.5k 4.8k 3.9k 3.5k 19k 19k landcover 0 0 0 0 0 0 0 9.7k 29k 85k 72k 81k 53k 30k 25k 85k mountain_peak 0 0 0 0 0 0 0 1.1k 1.8k 3.4k 4.4k 2.8k 1.4k 1.4k 869 4.4k water_name 0 0 0 0 0 0 0 0 0 486 461 433 452 1.2k 1.5k 1.5k aerodrome_label 0 0 0 0 0 0 0 0 0 0 666 289 273 221 221 666 aeroway 0 0 0 0 0 0 0 0 0 0 1.6k 2k 3k 3.3k 2.8k 3.3k poi 0 0 0 0 0 0 0 0 0 0 0 0 589 586 88k 88k building 0 0 0 0 0 0 0 0 0 0 0 0 0 59k 92k 92k housenumber 0 0 0 0 0 0 0 0 0 0 0 0 0 0 35k 35k full tile 7.9k 4k 9.5k 6.5k 3.7k 6.3k 21k 41k 84k 201k 184k 135k 114k 127k 253k 253k gzipped 6.2k 3.5k 7.1k 5.2k 3.1k 5k 14k 29k 60k 148k 137k 99k 84k 90k 161k 161k 0:01:10 DEB [archive] - Max tile: 253k (gzipped: 161k) 0:01:10 DEB [archive] - Avg tile: 5.5k (gzipped: 4.1k) using weighted average based on OSM traffic 0:01:10 DEB [archive] - # tiles: 4,115,031 0:01:10 DEB [archive] - # features: 5,738,475 0:01:10 INF [archive] - Finished in 21s cpu:1m17s avg:3.7 0:01:10 INF [archive] - read 1x(3% 0.6s wait:19s done:1s) 0:01:10 INF [archive] - encode 4x(57% 12s wait:2s done:1s) 0:01:10 INF [archive] - write 1x(20% 4s wait:14s) 0:01:10 INF [archive] - Finished in 1m11s cpu:3m57s gc:1s avg:3.3 0:01:10 INF [archive] - FINISHED! 0:01:10 INF [archive] - 0:01:10 INF [archive] - ---------------------------------------- 0:01:10 INF [archive] - data errors: 0:01:10 INF [archive] - render_snap_fix_input 16,942 0:01:10 INF [archive] - osm_multipolygon_missing_way 396 0:01:10 INF [archive] - osm_boundary_missing_way 55 0:01:10 INF [archive] - merge_snap_fix_input 15 0:01:10 INF [archive] - feature_polygon_osm_invalid_multipolygon_empty_after_fix 7 0:01:10 INF [archive] - feature_centroid_if_convex_osm_invalid_multipolygon_empty_after_fix 2 0:01:10 INF [archive] - omt_fix_water_before_ne_intersect 2 0:01:10 INF [archive] - render_snap_fix_input2 1 0:01:10 INF [archive] - omt_park_area_osm_invalid_multipolygon_empty_after_fix 1 0:01:10 INF [archive] - ---------------------------------------- 0:01:10 INF [archive] - overall 1m11s cpu:3m57s gc:1s avg:3.3 0:01:10 INF [archive] - lake_centerlines 2s cpu:6s avg:2.6 0:01:10 INF [archive] - read 1x(21% 0.5s done:2s) 0:01:10 INF [archive] - process 4x(0% 0s done:2s) 0:01:10 INF [archive] - write 1x(0% 0s done:2s) 0:01:10 INF [archive] - water_polygons 15s cpu:42s avg:2.7 0:01:10 INF [archive] - read 1x(42% 7s done:7s) 0:01:10 INF [archive] - process 4x(26% 4s wait:4s done:6s) 0:01:10 INF [archive] - write 1x(3% 0.5s wait:10s done:5s) 0:01:10 INF [archive] - natural_earth 7s cpu:15s avg:2.2 0:01:10 INF [archive] - read 1x(95% 6s) 0:01:10 INF [archive] - process 4x(13% 0.9s wait:6s) 0:01:10 INF [archive] - write 1x(0% 0s wait:7s) 0:01:10 INF [archive] - osm_pass1 2s cpu:8s avg:3.4 0:01:10 INF [archive] - read 1x(2% 0s wait:2s) 0:01:10 INF [archive] - parse 4x(35% 0.8s) 0:01:10 INF [archive] - process 1x(70% 2s) 0:01:10 INF [archive] - osm_pass2 21s cpu:1m23s avg:4 0:01:10 INF [archive] - read 1x(0% 0s wait:13s done:8s) 0:01:10 INF [archive] - process 4x(74% 16s) 0:01:10 INF [archive] - write 1x(2% 0.5s wait:20s) 0:01:10 INF [archive] - ne_lakes 0s cpu:0s avg:0 0:01:10 INF [archive] - boundaries 0s cpu:0s avg:0 0:01:10 INF [archive] - agg_stop 0s cpu:0s avg:0 0:01:10 INF [archive] - sort 1s cpu:4s avg:2.7 0:01:10 INF [archive] - worker 1x(50% 0.7s) 0:01:10 INF [archive] - archive 21s cpu:1m17s avg:3.7 0:01:10 INF [archive] - read 1x(3% 0.6s wait:19s done:1s) 0:01:10 INF [archive] - encode 4x(57% 12s wait:2s done:1s) 0:01:10 INF [archive] - write 1x(20% 4s wait:14s) 0:01:10 INF [archive] - ---------------------------------------- 0:01:10 INF [archive] - archive 108MB 0:01:10 INF [archive] - features 295MB -rw-r--r-- 1 runner runner 89M Feb 28 13:49 run.jar

Full logs: https://github.com/onthegomap/planetiler/actions/runs/22521942841

[msbarry]

this was the culprit - trying to allocate slabs of memory that were too big and the JVM couldn't find space for them.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
90.9% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud