Skip to content
This repository has been archived by the owner on Oct 2, 2024. It is now read-only.

New Machine Learning Model Extension Version 2.0.alpha schema and (de)serialization, validation package #2

Merged
merged 113 commits into from
Apr 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
113 commits
Select commit Hold shift + click to select a range
4d3955f
python package, cli, and old ml model spec validation
rbavery Dec 10, 2023
0aa94fc
refactor models and replace data object with common metadata band object
rbavery Dec 15, 2023
d67ca11
basic pydantic models for refactored model extension
rbavery Dec 16, 2023
195a07b
readme updates
rbavery Jan 5, 2024
9b3ad07
start filling out base models with schema described in README and mai…
rbavery Jan 5, 2024
f2ccf4c
mostly finish filling out object models
rbavery Jan 6, 2024
e81bee9
some changes to fields and language edits
rbavery Jan 6, 2024
9645354
poetry run stac-model generates json example
rbavery Jan 6, 2024
cdc6cba
README updates
rbavery Jan 6, 2024
f746d6c
add to CHANGELOG
rbavery Jan 6, 2024
c68dd72
address comments
rbavery Jan 9, 2024
b5a1fc8
address more comments
rbavery Jan 9, 2024
ffa398e
address most first draft comments
rbavery Jan 10, 2024
0716ee3
add container instructions
rbavery Jan 10, 2024
ab74151
fix fields missing object, add accelerator constrained field to runtime
rbavery Jan 10, 2024
eb8b80a
account for models that take parameters with one or more of each inpu…
rbavery Jan 10, 2024
5378bae
properly escape or operators
rbavery Jan 10, 2024
0c9f1f2
language edits for model input, change stats to have type option for …
rbavery Jan 10, 2024
b2cc2f0
language edits down to Result Array Object, specify derived_from rel …
rbavery Jan 10, 2024
f72faaa
update Changelog, more language edits
rbavery Jan 10, 2024
1f1891b
update Changelog
rbavery Jan 10, 2024
71cac72
link lots of fields and objects to readme text within tables
rbavery Jan 10, 2024
03e334c
new precommit, align with draft 2 of spec
rbavery Jan 12, 2024
1e58c02
version update
rbavery Jan 12, 2024
f379125
precommit version update
rbavery Jan 12, 2024
cd628c1
use classification extension instead of custom class map object
rbavery Feb 14, 2024
3602800
flatten architecture object into top level fields, use classification…
rbavery Feb 15, 2024
c0946d1
add best practices doc referencing processing extension
rbavery Feb 15, 2024
2ba7483
refer to best practices in readme
rbavery Feb 15, 2024
d4c8f3f
add processing ex
rbavery Feb 15, 2024
d7d99be
make task enum searchable, add to top level, keep in output object
rbavery Feb 15, 2024
4310971
update Model Input object to account for normalization with clipping …
rbavery Feb 15, 2024
b95a696
remove superflous data type field, rely on data type in the array obj…
rbavery Feb 15, 2024
af98031
update stac_model and example
rbavery Feb 15, 2024
1aba5fc
rescale -> resize add super res task
rbavery Feb 15, 2024
f86a64b
update example
rbavery Feb 15, 2024
76c0ba9
best practices for processing ext, format and lint
rbavery Feb 23, 2024
b8efda0
address metadata and text comments
rbavery Feb 23, 2024
fb1de05
remove old model_metadata.py
rbavery Feb 23, 2024
b2318ad
correct mlm: prefix
rbavery Feb 23, 2024
43a7440
remove extra column
rbavery Feb 23, 2024
7a25d92
move stac_model up with gh actions, readme, templates
rbavery Feb 24, 2024
54a52fa
update test, make an examples module
rbavery Feb 26, 2024
41cf8ea
simplify test
rbavery Feb 26, 2024
21d1aa9
increment stac model version
rbavery Feb 26, 2024
2e07901
optional annotations, downgrade pydantic
rbavery Feb 26, 2024
3178b27
Merge branch 'crim-ca:main' into validate
rbavery Feb 27, 2024
2b62d7b
combine stac_model and pystac metadata
rbavery Feb 28, 2024
4590141
produce pystac item in example but can't serialize datetime
rbavery Feb 28, 2024
4fc2e8e
remove helper and use pystac.Item in example
rbavery Feb 28, 2024
30269d4
update cli example. still getting datetime serialization issue
rbavery Feb 28, 2024
9ddff24
export an example with stac common metadata, derived from link to dat…
rbavery Feb 28, 2024
0bed29b
remove mlm_prefix in pydantic models
rbavery Mar 7, 2024
bf3b07f
address comments
rbavery Mar 7, 2024
c52daa7
update poetry, remove s3Path since it fails with recent pydantic and …
rbavery Mar 8, 2024
c7c75ca
roles for asset objects
rbavery Mar 8, 2024
d091c2e
specify how to use commit hash and add to example
rbavery Mar 8, 2024
9e58fdf
fields reordered so datetimes are together
rbavery Mar 8, 2024
f9f66d6
remove geometry models
rbavery Mar 8, 2024
406279c
add roles
rbavery Mar 10, 2024
90a63d4
changelog updates
rbavery Mar 20, 2024
42bbebb
address feedback on formatting and descriptions
rbavery Mar 20, 2024
3429dea
reorg runtime fields upward and remove runtime object
rbavery Mar 20, 2024
2570e62
add asset descriptions
rbavery Mar 20, 2024
fbdb482
move some non-search info to assets
rbavery Mar 20, 2024
aa3bc9b
linking and formatting
rbavery Mar 20, 2024
2a2039b
remove parameters, add artifact type field
rbavery Mar 26, 2024
67b4688
[wip] address PR comments about tasks definitions
fmigneault-crim Mar 28, 2024
efe223b
apply PR recommendations
fmigneault-crim Mar 29, 2024
c79ea01
add best practice details
fmigneault-crim Mar 29, 2024
4d765c2
add yet again more best practices to integrate other STAC extensions
fmigneault-crim Mar 29, 2024
4db3b94
more best practices (relates to https://github.com/stac-extensions/cl…
fmigneault-crim Mar 30, 2024
669c9a3
adjustments from PR review
fmigneault-crim Mar 30, 2024
edcc8a2
add more mlm:accelerator details (relates to https://github.com/crim-…
fmigneault Mar 30, 2024
06ee0ef
add details about link releation types
fmigneault Mar 30, 2024
1a50057
add details about dimensions and tasks
fmigneault Mar 30, 2024
1faf4d9
more examples and details
fmigneault Apr 2, 2024
501971a
[wip] updating JSON-schema with MLM fields
fmigneault Apr 2, 2024
6ec1cd5
[wip] more updates to JSON schema for MLM definitions
fmigneault Apr 4, 2024
8aca9b3
more schema adjustments
fmigneault Apr 4, 2024
ab41765
more details about expected values for dim_order + pretrained flag
fmigneault Apr 4, 2024
be58e86
address incompatibility of 'end_datetime=null' with STAC Core (relate…
fmigneault Apr 4, 2024
8b46388
add mlm:hyperparameters defintion (fixes https://github.com/crim-ca/d…
fmigneault Apr 4, 2024
2b87297
add example bands and statitics details
fmigneault Apr 4, 2024
269bd73
update pydantic models with new json-schema fields
fmigneault Apr 4, 2024
03e7e06
add details & example with 'eo:bands' for special JSON schema conside…
fmigneault Apr 4, 2024
2d6c70b
update examples working against JSON schema (except check for cross-b…
fmigneault Apr 5, 2024
d111678
adjust pydantic eurosat_example with json-schema fields
fmigneault Apr 5, 2024
4d57e41
fix pydantic drop unset fields as intended
fmigneault Apr 5, 2024
4eb30da
add OmitIfNone reference code
fmigneault Apr 5, 2024
2155745
fix invalid raster/eo bands/statistics definitions in examples
fmigneault Apr 5, 2024
9d14ac6
update schema title and description
fmigneault Apr 9, 2024
afe0a9a
remove out of date items from changelog
rbavery Apr 9, 2024
1fb5f21
include PR recommended changes
fmigneault Apr 11, 2024
f1bee68
fix github ci command to instlal poetry
fmigneault Apr 17, 2024
5ad13fa
update ci commands
fmigneault Apr 17, 2024
a6bf8ee
update and fix markdown linting
fmigneault Apr 17, 2024
feb2ce0
fix missing jsonschema dependency
fmigneault Apr 17, 2024
8c62744
fix typing definitions
fmigneault Apr 18, 2024
5de6693
fix pydantic recursion error on JSON type
fmigneault Apr 18, 2024
7f7620c
more linting fixes
fmigneault Apr 18, 2024
1d7d17a
ignore for remark-lint
fmigneault Apr 18, 2024
a1872e8
add remark-lint ignore to npm scripts
fmigneault Apr 18, 2024
7f16176
downgrade remark-gfm
fmigneault Apr 18, 2024
1a5927e
drop remark-gfm causing issues
fmigneault Apr 18, 2024
a1192d3
update node in CI and reapply remark-gfm
fmigneault Apr 18, 2024
43b8691
fix STAC examples linting
fmigneault Apr 18, 2024
7d95cca
fix STAC MLM examples - remove old (invalid) DLM examples
fmigneault Apr 18, 2024
728dcba
fix STAC object self-references in python tests
fmigneault Apr 18, 2024
ead9833
add Python 3.12 to CI + rename CI to be more representative than 'build'
fmigneault Apr 18, 2024
7ef029d
fix incorrectly interpretation of pydoclint exclude dirs
fmigneault Apr 18, 2024
b1804fc
remove unnecessary package with dependency flagged by safety
fmigneault Apr 18, 2024
f2b7dc2
remove unnecessary package with dependency flagged by safety
fmigneault Apr 18, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
# Git
.git
.gitignore
.github

# Docker
.dockerignore

# IDE
.idea
.vscode

# Byte-compiled / optimized / DLL files
__pycache__/
**/__pycache__/
*.pyc
*.pyo
*.pyd
.Python
*.py[cod]
*$py.class
.pytest_cache/
..mypy_cache/

# poetry
.venv
requirements*.txt

# C extensions
*.so

# Virtual environment
.venv
venv

.DS_Store
.AppleDouble
.LSOverride
._*
24 changes: 24 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Check http://editorconfig.org for more information
# This is the main config file for this project:
root = true

[*]
charset = utf-8
end_of_line = lf
insert_final_newline = true
indent_style = space
indent_size = 2
trim_trailing_whitespace = true

[*.{py, pyi}]
indent_style = space
indent_size = 4

[Makefile]
indent_style = tab

[*.md]
trim_trailing_whitespace = false

[*.{diff,patch}]
trim_trailing_whitespace = false
17 changes: 17 additions & 0 deletions .github/.stale.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Number of days of inactivity before an issue becomes stale
daysUntilStale: 120
# Number of days of inactivity before a stale issue is closed
daysUntilClose: 30
# Issues with these labels will never be considered stale
exemptLabels:
- pinned
- security
# Label to use when marking an issue as stale
staleLabel: stale
# Comment to post when marking an issue as stale. Set to `false` to disable
markComment: >
This issue has been automatically marked as stale because it has not had
recent activity. It will be closed if no further activity occurs in 30 days. Thank you
for your contributions.
# Comment to post when closing a stale issue. Set to `false` to disable
closeComment: false
36 changes: 36 additions & 0 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
---
name: ":bug: Bug report"
about: If something isn't working 🔧
title: ''
labels: bug, needs-triage
assignees:
---

## :bug: Bug Report

<!-- A clear and concise description of what the bug is. -->

## :microscope: How To Reproduce

Steps to reproduce the behavior:

1. ...

### Code sample

<!-- If applicable, attach a minimal code sample to reproduce the described issue. -->

### Environment

* OS: [e.g. Linux / Windows / macOS]
* Python version
* stac-model version


## :chart_with_upwards_trend: Expected behavior

<!-- A clear and concise description of what you expected to happen. -->

## :paperclip: Additional context

<!-- Add any other context about the problem here. -->
3 changes: 3 additions & 0 deletions .github/ISSUE_TEMPLATE/config.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Configuration: https://help.github.com/en/github/building-a-strong-community/configuring-issue-templates-for-your-repository

blank_issues_enabled: false
23 changes: 23 additions & 0 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
name: ":rocket: Feature request"
about: Suggest an idea for this project 🏖
title: ''
labels: enhancement, needs-triage
assignees:
---

## :rocket: Feature Request

<!-- A clear and concise description of the feature proposal. -->

## :sound: Motivation

<!-- Please describe the motivation for this proposal, such as the problem it solves, the intended users for the feature and what does success look like and how we can measure that. -->

## :satellite: Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered. -->

## :paperclip: Additional context

<!-- Add any other context or screenshots about the feature request here. -->
28 changes: 28 additions & 0 deletions .github/ISSUE_TEMPLATE/question.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
---
name: "red_question_mark Question"
about: Ask a question about this project 🎓
title: ''
labels: question, needs-triage
assignees:
---

## Checklist

<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->

- [ ] I've searched the project's [`issues`][1], looking for the following terms:
- [...]

## :question: Question

<!-- What is your question -->

How can I [...]?

Is it possible to [...]?

## :paperclip: Additional context

<!-- Add any other context or screenshots about the feature request here. -->

[1]: https://github.com/crim-ca/stac-model/issues
30 changes: 30 additions & 0 deletions .github/PULL_REQUEST_TEMPLATE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
## Description

<!-- Add a more detailed description of the changes if needed. -->

## Related Issue

<!-- If your PR refers to a related issue, link it here. -->

## Type of Change

<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->

- [ ] :books: Examples, docs, tutorials or dependencies update;
- [ ] :wrench: Bug fix (non-breaking change which fixes an issue);
- [ ] :clinking_glasses: Improvement (non-breaking change which improves an existing feature);
- [ ] :rocket: New feature (non-breaking change which adds functionality);
- [ ] :boom: Breaking change (fix or feature that would cause existing functionality to change);
- [ ] :closed_lock_with_key: Security fix.

## Checklist

<!-- Mark with an `x` all the checkboxes that apply (like `[x]`) -->

- [ ] I've read the [`CONTRIBUTING.md`][2] guide;
- [ ] I've updated the code style using `make codestyle`;
- [ ] I've written tests for all new methods and classes that I created;
- [ ] I've written the docstring in `Google` format for all the methods and classes that I used.

[1]: https://github.com/rbavery/stac-model/blob/master/CODE_OF_CONDUCT.md
[2]: https://github.com/rbavery/stac-model/blob/master/CONTRIBUTING.md
35 changes: 35 additions & 0 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Configuration: https://dependabot.com/docs/config-file/
# Docs: https://docs.github.com/en/github/administering-a-repository/keeping-your-dependencies-updated-automatically

version: 2

updates:
- package-ecosystem: "pip"
directory: "/"
schedule:
interval: "monthly"
allow:
- dependency-type: "all"
commit-message:
prefix: ":arrow_up:"
open-pull-requests-limit: 5

- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "monthly"
allow:
- dependency-type: "all"
commit-message:
prefix: ":arrow_up:"
open-pull-requests-limit: 5

- package-ecosystem: "docker"
directory: "/docker"
schedule:
interval: "monthly"
allow:
- dependency-type: "all"
commit-message:
prefix: ":arrow_up:"
open-pull-requests-limit: 5
28 changes: 28 additions & 0 deletions .github/release-drafter.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Release drafter configuration https://github.com/release-drafter/release-drafter#configuration
# Emojis were chosen to match the https://gitmoji.dev/

name-template: "v$NEXT_PATCH_VERSION"
tag-template: "v$NEXT_PATCH_VERSION"

categories:
- title: ":rocket: Features"
labels: [enhancement, feature]
- title: ":wrench: Fixes & Refactoring"
labels: [bug, refactoring, bugfix, fix]
- title: ":package: Build System & CI/CD"
labels: [build, ci, testing]
- title: ":boom: Breaking Changes"
labels: [breaking]
- title: ":memo: Documentation"
labels: [documentation]
- title: ":arrow_up: Dependencies updates"
labels: [dependencies]

template: |
## What's Changed

$CHANGES

## :busts_in_silhouette: List of contributors

$CONTRIBUTORS
9 changes: 6 additions & 3 deletions .github/remark.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,18 @@ plugins:
# Apply some recommended defaults for consistency
- remark-preset-lint-consistent
- remark-preset-lint-recommended
- lint-no-html
- - lint-no-html
- false
# General formatting
- - remark-lint-emphasis-marker
- '*'
- remark-lint-hard-break-spaces
- remark-lint-blockquote-indentation
- remark-lint-no-consecutive-blank-lines
- - remark-lint-maximum-line-length
- 150
- 120
# GFM - autolink literals, footnotes, strikethrough, tables, tasklist
- remark-gfm
# Code
- remark-lint-fenced-code-flag
- remark-lint-fenced-code-marker
Expand All @@ -37,7 +40,7 @@ plugins:
- - remark-lint-unordered-list-marker-style
- '-'
- - remark-lint-list-item-indent
- space
- space
# Tables
- remark-lint-table-pipes
- remark-lint-no-literal-urls
16 changes: 16 additions & 0 deletions .github/workflows/greetings.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Greetings

on: [pull_request, issues]

jobs:
greeting:
runs-on: ubuntu-latest
steps:
- uses: actions/first-interaction@v1
with:
repo-token: ${{ secrets.GITHUB_TOKEN }}
pr-message: 'Hello @${{ github.actor }}, thank you for submitting a PR!'
issue-message: |
Hello @${{ github.actor }}, thank you for submitting an issue!

If this is a bug report, please provide **minimum viable code to reproduce your issue**.
16 changes: 16 additions & 0 deletions .github/workflows/release-drafter.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
name: Release Drafter

on:
push:
# branches to consider in the event; optional, defaults to all
branches:
- main

jobs:
update_release_draft:
runs-on: ubuntu-latest
steps:
# Drafts your next Release notes as Pull Requests are merged into "master"
- uses: release-drafter/[email protected]
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
36 changes: 36 additions & 0 deletions .github/workflows/stac-model.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Check Python Linting and Tests

on: [push, pull_request]

jobs:
stac-model:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.10", "3.11", "3.12"]

steps:
- uses: actions/checkout@v2
- name: Set up Python ${{ matrix.python-version }}
uses: actions/[email protected]
with:
python-version: ${{ matrix.python-version }}

- name: Install poetry
run: make poetry-install

- name: Set up cache
uses: actions/[email protected]
with:
path: .venv
key: venv-${{ matrix.python-version }}-${{ hashFiles('pyproject.toml') }}-${{ hashFiles('poetry.lock') }}
- name: Install dependencies
run: make install-dev

- name: Run checks
run: |
make lint-all

- name: Run tests
run: |
make test
7 changes: 5 additions & 2 deletions .github/workflows/test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,13 @@ jobs:
runs-on: ubuntu-latest
strategy:
matrix:
node-version: [14.x]
node-version: [16]
steps:
- uses: actions/setup-node@v1
with:
node-version: ${{ matrix.node-version }}
cache: npm
- uses: actions/checkout@v2
- run: |
npm install
npm test
npm test
Loading
Loading