-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.qmd
99 lines (66 loc) · 6.76 KB
/
index.qmd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
---
title: "Cloud-Optimized Geospatial Formats Guide"
subtitle: "Methods for Generating and Testing Cloud-Optimized Geospatial Formats"
---
## Why Cloud Optimize?
Geospatial data is experiencing exponential growth in both size and complexity. As a result, traditional data access methods, such as file downloads, have become increasingly impractical for achieving scientific objectives. With the limitations of these older methods becoming more apparent, cloud-optimized geospatial formats present a much-needed solution.
Cloud optimization enables efficient, on-the-fly access to geospatial data, offering several advantages:
1. **Reduced Latency**: Subsets of the raw data can be fetched and processed much faster than downloading files.
2. **Scalability**: Cloud-optimized formats are usually stored on cloud object storage, which is infinitely scalable. Object storage supports many parallel read requests when combined with metadata about where different data bits are stored, making it easier to work with large datasets.
3. **Flexibility**: Cloud-optimized formats allow for high levels of customization, enabling users to tailor data access to their specific needs. Additionally, advanced query capabilities provide the freedom to perform complex operations on the data without downloading and processing entire datasets.
4. **Cost-Effectiveness**: Reduced data transfer and storage needs can lower costs. Many of these formats offer compression options, which reduce storage costs.
If you want to provide optimized access to geospatial data, this guide is designed to help you understand the best practices and tools available for cloud-optimized geospatial formats.
## Built for the Community, by the Community
There is no one-size-fits-all approach to cloud-optimized data. Still, the community has developed many tools for creating and assessing geospatial formats that should be organized and shared.
This guide provides the landscape of cloud-optimized geospatial formats and the best-known answers to common questions.
## How to Get Involved
Read the [Get Involved](./contributing.qmd) page if you want to contribute or modify content.
If you have a question or idea for this guide, please start a [Github Discussion](https://github.com/cloudnativegeo/cloud-optimized-geospatial-formats-guide/discussions/new/choose).
## The Opportunity
Storing data in the cloud does not, on its own, solve geospatial's data problems. Users cannot reasonably wait to download, store, and work with large files on their machines. Large volumes of data must be available via subsetting methods to access data in memory.
While it is possible to provide subsetting as a service, this requires ongoing maintenance of additional servers and extra network latency when accessing data (data has to go to the server where the subsetting service is running and then to the user). With cloud-optimized formats and the appropriate libraries, subsets of data can be accessed directly from an end user's machine without introducing an additional server.
Regardless, users will access data over a network, which must be considered when designing the cloud-optimized format. Traditional geospatial formats are optimized for on-disk access via small internal chunks. A network introduces latency, and the number of requests must be considered.
As a community, we have arrived at the following **cloud-optimized format pattern:**
1. Metadata includes addresses for data blocks.
2. Metadata is stored in a consistent format and location.
3. Metadata can be read once.
4. Metadata can read the underlying file format, which supports subsetted access via addressable chunks, internal tiling, or both.
These characteristics allow for parallelized and partial reading.
## Data Type to Traditional to Cloud-Optimized Geospatial File Format Table
The diagram below depicts how some of the cloud-optimized formats discussed in this guide are cloud-optimized formats of traditional geospatial file formats.
Notes:
- Some data formats cover multiple data types, specifically:
- GeoJSON can be used for vector and point cloud data.
- HDF5 can be used for point cloud data or data cubes (or both via groups).
- GeoParquet and FlatGeobuf can be used for vector data or point cloud data.
- LAS files are intended for 3D points, not 2D points (since COPC files are compressed LAS files, the same goes for COPC files).
- [TopoJSON](https://github.com/topojson/topojson) (an extension of GeoJSON that encodes topology) and [newline-delimited GeoJSON](https://stevage.github.io/ndgeojson/) are types of GeoJSON worth mentioning but have yet to be explicitly represented in the diagram.
- GeoTIFF and GeoParquet are geospatial versions of the non-geospatial file formats TIFF and Parquet, respectively. FlatGeobuf builds upon the non-geospatial [flatbuffers](https://github.com/google/flatbuffers) serialization library (though flatbuffers is not a standalone file format).
## Table of Contents
1. [Overview of Formats (slideshow)](./overview.qmd)
2. Formats
a. [Cloud-Optimized GeoTIFFs](./cloud-optimized-geotiffs/intro.qmd)
b. [Zarr](./zarr/intro.qmd)
c. [Kerchunk](./kerchunk/intro.qmd)
d. [Cloud-Optimized NetCDF4/HDF5](./cloud-optimized-netcdf4-hdf5/index.qmd)
e. [Cloud-Optimized Point Clouds (COPS)](./copc/index.qmd)
f. [GeoParquet](./geoparquet/index.qmd)
g. [FlatGeobuf](./flatgeobuf/intro.qmd)
h. [PMTiles](./pmtiles/intro.qmd)
3. [Cookbooks](./cookbooks/index.qmd)
## Running Examples
Most of the data formats covered in this guide have a Jupyter Notebook example that covers the basics of reading and writing the given format. At the top of each notebook is a link to an environment.yml file describing what libraries must be installed to run correctly. You can use [Conda](https://www.anaconda.com/download) or [Mamba](https://mamba.readthedocs.io/en/latest/index.html) (a successor to Conda with faster package installs) to install the environment needed to run the notebook.
## Authors
* [Aimee Barciauskas](https://developmentseed.org/team/aimee-barciauskas)
* [Alex Mandel](https://developmentseed.org/team/alex-mandel)
* [Kyle Barron](https://github.com/kylebarron)
* [Zac Deziel](https://developmentseed.org/team/zac-deziel)
* [Overview Slide](./overview.qmd) credits: Vincent Sarago, Chris Holmes, Patrick Quinn, Matt Hanson, Ryan Abernathey
## Questions to Ask When Generating Cloud-Optimized Geospatial Data in Any Format
1. What variable(s) should be included in the new data format?
2. Will you create copies to optimize for different needs?
3. What is the intended use case or usage profile? Will this product be used for visualization, analysis, or both?
4. What is the expected access method?
5. How much of your data is typically rendered or selected at once?
{{< include _thankyous.qmd >}}