Managing functionality in the ODC ecosystem - exposing valuable tools for data manipulation and analysis

[caitlinadams]

Context

I've been thinking about how to expose a lot of the valuable tools that have been created as part of Digital Earth Australia and Digital Earth Africa. At the moment, the tools for these platforms are independently maintained by both parties. I think there could be a lot of benefits to be gained by finding a way to generalise them so that they could be used by any project employing the ODC style of working with xarray. For example, some functions are

• load_ard
• calculate_indices
• xr_vectorize
• temporal_statistics

The Challenge

But where to put them? We have two odc- packages that provide additonal functionality beyond datacube core:

• odc-stac: load data from STAC in an ODC-style xarray
• odc-geo: manage shape and geo information, and provide an odc accessor for xarray

There is also odc-tools and odc-aglo; odc-tools appears to be for development/testing of functionality, and odc-algo isn't wholy independent of datacube-core, as discussed here: opendatacube/odc-algo#7

There is the challenge of the DE Aus/Africa functions being a mix of data/product dependent (load_ard and calculate_indices) and data independent (xr_vectorize and temporal_statistics). I think the data dependent functions could be handled by having a more generic function, which is then customised using platform specific configuration -- something I'm actively exploring but don't have a solution for yet.

Discussion

My question is really, "What should go where?". This arose for me after reading opendatacube/odc-geo#117, which discusses the inclusion of opinionated masking into odc-geo.

I'm not 100% convinced that opinionated masking does belong in odc-geo, as it's not specifically related to handling shape and geometry, but more related to data manipulation. But it is a valuable and useful function, as indicated by the high use of the two load_ard functions by Digital Earth Australia and Digital Earth Africa, both of which provide opionated, product-based cloud masking.

But if not odc-geo, where should opinionated masking go? What about other data-independent manipulation functions? I'm curious about the community's thoughts on this, and whether there's interest from those using ODC to develop a datacube independent package like odc-geo, but focussed more on data manipulation and analysis. Perhaps this is what odc-algo could be if decoupled from its dependence on datacube.

[robbibt]

Thanks for starting this discussion off @caitlinadams - I think it's a really important question! My thoughts on a possible heirarchy of functionality would go something like this:

• odc-geo: For completely generic/dataset agnostic tools for manipulating georeferenced xarray data, regardless of what kind of data they contain. So high-level tools like reprojecting, generic masking functionality (e.g. tools for creating/applying/cleaning masks, but not opinionated masking configurations or anything specific to any individual product/scientific fields), generic plotting (e.g. basic RGB plots without any opinionated defaults) etc.
• <potential datacube independent tools package>: For generic EO/earth science-related tools for manipulating data loaded from both datacube and odc-stac. This could include more opinionated code designed to make it easier to use common EO datasets (e.g. Landsat/Sentinel-2-specific masking tools), as well as domain-specific tools like pansharpening, band indices etc. Ideally, this package would make use of higher level functionality from odc-geo wherever possible, and would not be specific to individual platforms - e.g. users wouldn't need the DEA or DE Africa Sandbox to run these tools.
• dea-tools, dea-africa-tools: For platform-specific/bespoke tooling for DEA and DE Africa, including highly opinionated functions or configurations that only run on DEA or DE Africa software or products. Ideally, these packages would be as minimal as we can make them - anything more generic would be better placed in <potential datacube independent tools package> (if EO/earth science specific) or odc-geo (if applicable to any georeferenced xarray data).

odc.algo/odc.tools are trickier - I haven't worked out where these fit into the framework above, largely because they contain such a grab-bag mix of new/old/cutting edge/outdated/experimental functionality (I do think though that there is a lot of content in there that would be easier to access and more discoverable in either odc-geo or a <potential datacube independent tools package>).

(I guess on a personal note, I've found odc-geo pretty revolutionary for my work (especially due to the .odc accessor) - even though it's got "geo" in the name, I see it more as a package of highly generic data manipulation tools for analysing xarray data vs. a purely spatial package. But I think there's definitely a huge opportunity/niche for something sitting in the middle, combining those high level tools in ways that make them generically useful for EO/earth science users. 🙂 )

[SpacemanPaul]

It is also worth discussing where datacube-core sits with respect to all this, noting that core will depend on (and use) odc-geo from the 1.9.0 release.

Depends-on-core vs doesn't-depend-on-core makes a big difference for ease of installation
(and meanwhile in odc-geo Russia, core-depends-on-YOU!)

[Kirill888]

I don't think we as a species have figured out THE CORRECT WAY of partitioning software into modules, it's probably somewhere between C++ (let's just implement everything from scratch and not use any external dependencies, cause they are hard, e.g. BOOST, QT) and NPM (huge number of extremely tiny modules, many unmaintained ready for hostile takeover, leftpad being just one famous example).

datacube-core has grown into a giant kitchen-sink at the time when odc-tools repo was spun out. From odc-tools all the other odc-* sub-modules have sprung up. The main reason for that split was to enable delivery of the new code to users, it was simply too hard to make a release of datacube-core, and without proper release it's hard to get feedback from the users.

Making datacube release is hard because: (A) it is being used (which is nice) and (B) because every release potentially introduces database migration, and nobody wants to do that, and for a good reason. All the tricky installation requirements are also mainly from the database layer, and to lesser extent due to netcdf support being non-optional (fixed now?).

I would advocate moving towards a properly separate database layer, and properly separate data load layer. As these are the two places that require most testing from users during migrations.

We should also think about makingpsycopg2 dependency optional, so that people CAN use psycopg2-binary if they choose, or not install it at all if they are using some other db back-end, or not even connecting to any database and just wanting to use some stand-alone code from datacube.*. But that approach is not without downsides: guarding against possibly unavailable imports is hard, testing all possible configurations is hard, educating users about extra requirements is hard (will require combination of good docs and good error messages).