[addresses] Add addresses for Poland from government's open data

[ttomasz]

Category Feedback

Polish government releases a lot of data into public domain like buildings, addresses, etc.

I propose adding the address dataset into Overture. It's around 8 151 958 address points at the moment.

Data is in GML format but I have prepared sample code that parses that into geoparquet using Apache Spark. Code is available here: https://github.com/ttomasz/polish_addresses_spark_parser The repo is dual use and I generate dataset with one schema for polish community and second one can be used as a basis for Overture.

Geoparquet files created using code from repo above are available here for reference: https://data.openstreetmap.org.pl/addresses/

At the moment the poland_addresses.parquet file does not conform to the Overture address schema exactly but I can modify it to be compliant if there is interest in actually using that code. I can also explain any of the data attributes or code if needed.

Dependency with other categories, if any.

No response

[jwass]

Hi @ttomasz Thanks for this! On the addresses side of things we are pointing people to add new data sources to https://openaddresses.io rather than build a separate database of openly available address datasets.

Their data is here https://github.com/openaddresses/openaddresses/tree/master/sources/pl. If your data is different, would you be up for submitting a pull request to open addresses? Once it's in their format we can easily add it to Overture and I'd be happy to help facilitate that.

[jwass]

@ttomasz It looks like the scripts that download and process the OpenAddresses Poland data https://github.com/openaddresses/openaddresses/tree/master/scripts/pl are from several years ago. They are mainly processed from https://opendata.geoportal.gov.pl/prg/adresy/PRG-punkty_adresowe.zip. Does this differ from the GML file you're processing?

[ttomasz]

Well I was a bit hesitant to try adding to openaddresses.io as they lack documentation and basically say "look at json documents and figure it out" (at least that was my experience when I skimmed their repo). If you can guide me on how their stuff actually works then I could try to make a PR.

The data from the url that you sent is the same as what I was processing. Generally one can expect new addresses at least weekly in that dataset.

[ttomasz]

Though I see that they ignore a lot of attributes like voivodeship, county, municipality and they don't filter on status which means that addresses that are planned but not existing are also in this dataset.

[ttomasz]

Schema in openaddresses.io is kind of limited as it has only two levels of administrative units, city and street. While for most addresses it would probably be enough it would kind of be a shame to lose the data while some addresses have full info provided e.g.:

┌─────────┬────────────────────┬─────────────────────┬───────────────┬──────────────┬───────────┬───────────────┬─────────────┬─────────────┐
│ country │    voivodeship     │       county        │ municipality  │     city     │ city_part │    street     │ housenumber │ postal_code │
│ varchar │      varchar       │       varchar       │    varchar    │   varchar    │  varchar  │    varchar    │   varchar   │   varchar   │
├─────────┼────────────────────┼─────────────────────┼───────────────┼──────────────┼───────────┼───────────────┼─────────────┼─────────────┤
│ Polska  │ kujawsko-pomorskie │ golubsko-dobrzyński │ Golub-Dobrzyń │ Paliwodzizna │ Ruziec    │ Wiśniowa      │ 7           │ 87-400      │
│ Polska  │ kujawsko-pomorskie │ golubsko-dobrzyński │ Golub-Dobrzyń │ Paliwodzizna │ Ruziec    │ Brzoskwiniowa │ 4           │ 87-400      │
│ Polska  │ kujawsko-pomorskie │ golubsko-dobrzyński │ Golub-Dobrzyń │ Paliwodzizna │ Ruziec    │ Brzoskwiniowa │ 13          │ 87-400      │
│ Polska  │ kujawsko-pomorskie │ golubsko-dobrzyński │ Golub-Dobrzyń │ Paliwodzizna │ Ruziec    │ Brzoskwiniowa │ 11          │ 87-400      │
│ Polska  │ kujawsko-pomorskie │ golubsko-dobrzyński │ Golub-Dobrzyń │ Paliwodzizna │ Ruziec    │ Wiśniowa      │ 5           │ 87-400      │
└─────────┴────────────────────┴─────────────────────┴───────────────┴──────────────┴───────────┴───────────────┴─────────────┴─────────────┘

@jwass I don't get the relation between https://github.com/openaddresses/openaddresses/blob/master/sources/pl/dolnoslaskie.json and https://github.com/openaddresses/openaddresses/blob/master/scripts/pl/process_one.py at all. Can you explain why it seems like sources contain some links to cached files which looks like manual upload while the script to parse the data exists?

[jwass]

@ttomasz sorry for delayed response. I was out on vacation but back now.

Yeah - my understanding is OA needs a single file to convert to its output format. But if you have to do some other processing you put it in the scripts/ directory and then upload the result. But it seems like that doesn't work if you need to do it consistently. I can help ask over in the OA repo. If not, we should look into doing this on the Overture side.

For the Overture schema - we only want the parts of the admin hierarchy that are relevant to the address, rather than splitting into district, city, region, etc. We also chose the generic "address_levels" to avoid having to assign names to different levels for different countries which always seems to spur lots of debate. Based on https://www.upu.int/UPU/media/upu/PostalEntitiesFiles/addressingUnit/polEn.pdf it looks like Poland only needs a locality/city?

Also agree we should remove the planned addresses. Let's try to figure this out for Poland in Overture soon

[ttomasz]

@jwass Probably worth asking them how it works. On their webpage it suggests that they actively process new data so I don't know. If you already have converter from openaddress schema to overture schema it might be easier to just use their data.

As for the latter, well technically city, street, house number, and postal code form an unique address (there are some legitimately duplicated addresses but those are exceptions) but postal codes are not exactly super readable and it's kind of nice to have admin names. And address_levels seems to be designed to hold exactly this information.

Check out those queries that you can run with duckdb:

select *
from 'https://data.openstreetmap.org.pl/addresses/poland_addresses.parquet'
where place='Dąbrowa' and street = '' and housenumber = '43' order by administrative_units
;

select *
from 'https://data.openstreetmap.org.pl/addresses/poland_addresses.parquet'
where place='Nowa Wieś' and street = '' and housenumber = '3' order by administrative_units
; -- Nowa Wieś -> New Village in english, pretty popular name

select place, count(distinct administrative_units) as distinct_admin
from 'https://data.openstreetmap.org.pl/addresses/poland_addresses.parquet'
group by place
order by distinct_admin desc
limit 20
; -- There are many villages with the same names.