revise scope of the ml model extension to focus on model card and inference #15
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I'm starting a draft to refactor the extension to focus on making model inference easy, portable, and discoverable. This shifts the focus of the extension from serving inference, training, and retraining needs. Hopefully the spec will be simple and concrete enough that many providers and users of GeoML models will adopt it and develop tooling for it to make it easier to find and use models.
The big updates I think are needed:
rationale:
I think the extension as it currently stands is difficult to build on top of because it doesn't improve workflows for using ML models (either in training or inference). The spec doesn't enforce standards for how model artifacts need to be represented. Model creators might distribute the model runtime environment in many different ways (requirements.txt, setup.py. pyproject.toml, python package, conda, docker, no documentation), so seeking to capture and enforce this info in the spec seems too complex and like it wouldn't serve a large enough user base. I think it would be more helpful for more people if geo ML models had a detailed model card on performance and limitations, and to optionally point to a github repo with further details on the runtime environment for training.
For example, torchgeo captures model information, but these model weights are only tied to papers and source github repositories. there's no quick description of what dataset it is related to, what format the model artifacts are in, or what kind of model input and output relates to the weights: https://torchgeo.readthedocs.io/en/stable/api/models.html#resnet
I think the biggest "standards gap" is a metadata description for discovery and distribution of models for model users, rather than for model developers looking to train or retrain existing models.