Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

revise scope of the ml model extension to focus on model card and inference #15

Closed
wants to merge 1 commit into from

Conversation

rbavery
Copy link
Collaborator

@rbavery rbavery commented Nov 30, 2023

I'm starting a draft to refactor the extension to focus on making model inference easy, portable, and discoverable. This shifts the focus of the extension from serving inference, training, and retraining needs. Hopefully the spec will be simple and concrete enough that many providers and users of GeoML models will adopt it and develop tooling for it to make it easier to find and use models.

The big updates I think are needed:

  1. reducing the scope of the extension to describing and linking to inference requirements and a model card to describe other model characteristics.
  2. add input and output signatures for different modeling tasks to enable models to be run only by inspecting the metadata extension and setting up minimal dependencies (torch, torchvision)
  3. encourage use of compiled model formats (torch.export, onnx) with examples to increase focus on portability for GeoML models

rationale:

I think the extension as it currently stands is difficult to build on top of because it doesn't improve workflows for using ML models (either in training or inference). The spec doesn't enforce standards for how model artifacts need to be represented. Model creators might distribute the model runtime environment in many different ways (requirements.txt, setup.py. pyproject.toml, python package, conda, docker, no documentation), so seeking to capture and enforce this info in the spec seems too complex and like it wouldn't serve a large enough user base. I think it would be more helpful for more people if geo ML models had a detailed model card on performance and limitations, and to optionally point to a github repo with further details on the runtime environment for training.

For example, torchgeo captures model information, but these model weights are only tied to papers and source github repositories. there's no quick description of what dataset it is related to, what format the model artifacts are in, or what kind of model input and output relates to the weights: https://torchgeo.readthedocs.io/en/stable/api/models.html#resnet

I think the biggest "standards gap" is a metadata description for discovery and distribution of models for model users, rather than for model developers looking to train or retrain existing models.

@rbavery
Copy link
Collaborator Author

rbavery commented Dec 6, 2023

closing this in favor of https://github.com/crim-ca/dlm-extension

#13 (comment)

@rbavery rbavery closed this Dec 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant