Application: Maps + Geo Data on IPFS

[flyingzumwalt]

Work in progress - please contribute. See #40.

[spatialsparks]

Maybe this issue thread is worth mentioning here as a beginning: #25 ? (I know its more a demo, but came across it and thought it worth the mention)

[daviddias]

More notes on ipfs/notes#142

[olizilla]

I'd like to dust off this topic for some high-bandwidth research and discussion at the upcoming 2018 Q4 Lab Week. If you know anyone using IPFS for serving map tiles or geo data, please add a comment!

[daviddias]

Ping @mikolalysenko & @substack o/

https://github.com/mikolalysenko/hawaii-navigator

[daviddias]

Also, ping @davidar ✨

[mikolalysenko]

As a proof of concept hawaii-navigator shows how to build a decentralized routing index. You'll still need a geocoder and tile server to make it work well enough.

Might be fun to revisit this and try it out on some larger graphs to see how well it works.

[vmx]

Finally there's a recording of the talk I gave at the global FOSS4G 2018: https://youtu.be/rpx3kmpUwQI It contains a demo of storing raster tiles in IPFS.

[lidel]

https://wiki.openstreetmap.org/wiki/Planet.osm#IPFS:

A Weekly mirror of the continents-files from download.geofabrik.de is made available on IPFS at /ipns/ipfs.placebazaar.org, Accessible with any IPFS client, or via a gateway. For example https://ipfs.io/ipns/ipfs.placebazaar.org. A server is dedicated to seeding this data for a week.

[ryan-foamspace]

Hi everyone, I am the co-founder of FOAM a blockchain project building a consensus driven Map of the World and Proof of Location services.

Currently the FOAM Map is live map.foam.space. It operates as a Token Curated Registry for Point of Interest Data where tokens are staked when adding data, which could be challenged and open to a vote.

All of the metadata for each point is currently stored on IPFS and wanted to share with this thread.

Here is for example the transaction of a new point being added to the Map
https://etherscan.io/tx/0x70a88427976c1b0ca272ed3285aed8b1c25cc56a1aa7583f872134e17acf92cd#decodetab

And the IPFS hash
https://cloudflare-ipfs.com/ipfs/QmfMrMwfrmHrC28FrJkJyy5oek6qpKbkDsq6LK5ZtfJLg1

Only Point data is being curated at the moment and the base map being used is Mapbox/OSM. Having the tiles be served directly from IPFS is a next step

[renatrazumov]

this is a good idea

[am2222]

Hi,
I have a simple question related to this topic,
Assume we have out spatial data on ipfs networks. let's say a set of the point. To search these points based on a query window which options do we have?
I was thinking of a kind of Distributed R-tree for each file for fast data access and query. But wanted to know any other possible ideas.

[vmx]

@am2222 If you want to store only points which are lat/lon then you could also look into https://en.wikipedia.org/wiki/Geohash.

I personally prefer using R-trees for every geo related thing I do as they are more flexible as you can store any numeric data you want without defining any bounds up-front (it's about data vs. space partitioning data structures/algorithms)

[am2222]

@vmx thanks so much, Actually currently I have some methods to convert geo data into geohash but fast data access is the main issue for me now. Since the geo data can be at least line,point and polygon,
However about using R-trees for it do you recommend using R-tree structure for the file storage?something like this

`{
"root": [
[

  {
    "geohash": 1001,
    "adjacent": [
      "dg3:isea3h/11.25_58.2825/7/cell/947",
      "dg3:isea3h/11.25_58.2825/7/cell/974",
      "dg3:isea3h/11.25_58.2825/7/cell/975",
      "dg3:isea3h/11.25_58.2825/7/cell/1028",
      "dg3:isea3h/11.25_58.2825/7/cell/1029",
      "dg3:isea3h/11.25_58.2825/7/cell/1056"
    ],
    "centroid": "dg3:isea3h/11.25_58.2825/7/point/1001",
    "projection": "isea3h"
  }
],
 [
  {
    "geohash": 1002,
    "adjacent": [
      "dg3:isea3h/11.25_58.2825/7/cell/947",
      "dg3:isea3h/11.25_58.2825/7/cell/974",
      "dg3:isea3h/11.25_58.2825/7/cell/975",
      "dg3:isea3h/11.25_58.2825/7/cell/1028",
      "dg3:isea3h/11.25_58.2825/7/cell/1029",
      "dg3:isea3h/11.25_58.2825/7/cell/1056"
    ],
    "centroid": "dg3:isea3h/11.25_58.2825/7/point/1001",
    "projection": "isea3h"
  }
]

],
"type": "Feature",
"id": "dg3:isea3h/11.25_58.28252559/7/cell/1001",
"crs": {
"type": "name",
"properties": {
"name": "urn:ogc:def:crs:EPSG::4326"
}
}
}`
and make a data object based on a Tree structure and store spatial data in tree structure? I think in such way we can support window query, however we need to support knn queries as well.
I don't know about ability to update IPLD objects for example to add new feature to the tree.

[vmx]

@am2222 So far I haven't spent much time thinking about how to store R-trees in IPFS/IPLD. Coming from the database world, I currently think that local secondary indexes would be needed for fast retrieval. Though this would kind of work against the whole point of having everything stored distributed.

I would consider storing it in IPFS/IPLD an open problem :)

[am2222]

@vmx Actually I want to work in this field so I am looking for possible methods which I can look at. One it to look at the data storage as distributed geographical objects over network. But it needs a kind of secondary DHT I guess to find the data based on location and not based on hash values.
The second idea was to store them as a Feature sets, and save them in a tree based structure for fast data access, But I don't know if this is possible to do for IPLD data objects and let ipfs decides to store them on nodes instead of distributing them based on secondary DHT(even if it is possible )

[spatialsparks]

@am222 @vmx Im not really up to date with the whole IPFS debate, but I was wondering: is using an R-Tree or Quadtree or any dynamic spatial clustering system really suited for this type of data storage? Wouldnt predefined geo-hashes of some sort be more useful (albeit less flexible) here?

I was thinking something along the lines of Googles S2 algorithm or Ubers H3 hexagonal spatial indexing system. Then it would make sense to predefined the resolution you think is adequate. And the system you use should already have spatial information in the hash. I dont quite understand why you would want to store adjacent cells for example, the spatial index should have this in its hash? An example:

If I have the UberH3 hash of "891F83CAB6BFFFF" then this contains loads of information already: It contains the resolution, the centroid, the 6 edge vertecies (since its a hexagon) and all the neighbouring cells can be calculated quite easily (and not only direct neighbours but all neighbours in a distance of N rings). So you could have files with this index as its name and simple geojson within the file containing the exact feature information you need (e.g. coordinates, properties etc.).

Maybe looking into how vector tiles works (e.g. MapBox) would give some new ideas here?

Cheers

[vmx]

@JoranBeaufort I agree that those predefined hashing could make sense. Though I don't really have much experience with those as they were too limited for the use cases I had.

[am2222]

@JoranBeaufort @vmx
It is for a while that I started to work on this area (mostly on predefined geohashs (DGGG systems)) The reason that I am using "geohash": 1001, in my json format file above is because I aimed to use different geohash systems, However when it comes to storing points is becomes much easier but storing other data (raster, tin, network, polygon and line ) things get a bit complicated. Using R-tree structure to store data was to access and search data faster, but there are issues with updating and maintaining those data as well.
H3 uses an aperture 7 of hexagon grid which the parent and children does not overlap fully, I was aiming to use other systems.
About storing adjutant cells I was just thinking to provide a faster access to those, however based on different indexing methods for each DGG structure it is possible to find 6 neighbours and use them for local or focal spatial analysis. I probably try to remove them since it is possible to guess them and having them is unnecessary.

About vector tiles They work, but I believe they are mostly aimed for the fast vector data transformation for visualization purposes . and accessing them over IPFS is pretty strait forward as we just need to upload the whole vectortile folder on ipfs and read them based on xyz location of each requested tiles. But in term of geographical data sets we don't have data for all the tiles and we must be able to support spatial query over such data.

[lazyweirdo]

hello, is this question relevant here ?
https://discuss.ipfs.io/t/open-street-map-on-ipfs/6689
It is about adding tiles to ipfs, (dynamically if possible, not a requirement) ... it is a nice feature to have in the ecosystem, it would honor the 'P' of ipfs :)
What do you think ?

[lazyweirdo]

check this out https://github.com/chinanzio/ipfsmap , it is, maybe, useful here

[jessicaschilling]

Note: Discussion on applications of IPFS are happening over in the IPFS Forums now ... please continue the discussion there!

This issue is being moved over to the archived repo https://github.com/ipfs/apps/ for reference.