Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: Various improvements #13

Closed
wants to merge 209 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
209 commits
Select commit Hold shift + click to select a range
eef0203
Put test fils in `Tests` folder
candleindark May 20, 2024
69b0f96
Add `pydantic2linkml` package to project
candleindark May 20, 2024
fb1010f
Implement getting Pydantic core schemas of locally defined fields of …
candleindark May 23, 2024
4ff48c6
Implement func to ensure unique names of classes
candleindark May 24, 2024
75e210a
Outline the basic structure of the LinkML generator
candleindark May 25, 2024
76e052f
Ensure no name collection in initiating `LinkmlGenerator`
candleindark May 27, 2024
2a59078
Change the signature of `tools.ensure_unique_names`
candleindark May 27, 2024
dce98dd
Add all enums to the schema
candleindark May 28, 2024
7974869
Define func to normalized whitespace
candleindark May 28, 2024
1328950
Include `description` in `EnumDefinition`
candleindark May 28, 2024
23f1a05
Define interfaces to add slots and classes
candleindark May 28, 2024
eda50a7
Rearrange comments
candleindark May 28, 2024
dd4888c
Define internal map of models to their locally defined fields
candleindark May 28, 2024
8cfe73b
Implement `LinkmlGenerator._add_slots`
candleindark May 29, 2024
36f5cc6
Provide a function to bucketize items
candleindark May 30, 2024
3f783b0
Utilize `bucketize`
candleindark May 30, 2024
1d6a711
Add support of transforming values in `bucketize`
candleindark May 30, 2024
1607e0c
Adopt the new support in `bucketize`
candleindark May 30, 2024
f9025d2
Change slot generation to be handled by a class
candleindark May 31, 2024
58135c1
Implement `SlotGenerator.__init__`
candleindark May 31, 2024
22ef7b0
Outline `SlotGenerator.generate`
candleindark May 31, 2024
c515b05
Specifies interfaces of Pydantic core schema handler functions in `Sl…
candleindark May 31, 2024
bcfff9a
Reformat generate_linkml_from_aind.py with Black
candleindark May 31, 2024
af93dce
Remove unused imports
candleindark May 31, 2024
0423fb8
Replace use of `not` and `in` with `not in`
candleindark May 31, 2024
2f9b121
Remove unused variable in `generate_linkml_from_aind.main()`
candleindark May 31, 2024
be16880
Correct type annotation in `populate_enum` and `populate_basemodel`
candleindark May 31, 2024
674ad72
Move `get_all_modules()` to tools.py
candleindark May 31, 2024
ce5476e
Use inner function to simplify the interface of `get_all_modules()`
candleindark May 31, 2024
afeecc9
Implement `fetch_defs()` and provided needed tests
candleindark Jun 1, 2024
cb68232
Corret import statement for `CoreSchemaOrField`
candleindark Jun 2, 2024
c17f55e
Rename local variables
candleindark Jun 2, 2024
268700b
Correct retrieval of corresponding `core_schema.ModelSchema` of a Pyd…
candleindark Jun 2, 2024
87c7ef0
Remove limitation of not supporting multi inheritance in Pydantic models
candleindark Jun 3, 2024
989e5f0
Allow fetching of multi Pydantic parent models
candleindark Jun 3, 2024
477c204
Replace the use of `typing.Type` with `type`
candleindark Jun 3, 2024
7963980
Have `tools.fetch_defs()` returns tuple of sets instead of list
candleindark Jun 3, 2024
0ce4fc6
Update test for `tools.fetch_defs()` and remove unneeded wrapping class
candleindark Jun 3, 2024
a7ad7b0
Add test for instantiating `TestGenLinkml`
candleindark Jun 3, 2024
c177ccf
Assert `"model-field"` type in getting of field schema
candleindark Jun 3, 2024
ccd9397
Rename `SlotGenerator.build_schema_type_to_method` to `SlotGenerator.…
candleindark Jun 3, 2024
3499f64
Have `get_field_schema()` return `FieldSchema`
candleindark Jun 4, 2024
d9a1966
Add test for instantiation of `SlotGenerator`
candleindark Jun 4, 2024
5dcbaf5
Remove `add_defaults` param in `LinkmlGenerator.__init__()`
candleindark Jun 5, 2024
4310d20
Update `TestGenLinkml.test_instantiation_with_definitions_in_module`
candleindark Jun 6, 2024
decf92e
Attach corresponding `FieldInfo` to `FieldSchema`
candleindark Jun 7, 2024
766a3ea
Handle `None` arguments for `models` and `enums` in `LinkmlGenerator.…
candleindark Jun 7, 2024
32331fd
Establish supporting definitions in a generated schema
candleindark Jun 7, 2024
7abc950
Provide mechanism to slot generator to shape slot
candleindark Jun 8, 2024
354d2ee
Shape slot according to a `core_schema.AnySchema`
candleindark Jun 8, 2024
1f0e7cc
Extract the definition of the LinkML Any type to module level
candleindark Jun 8, 2024
b949282
Rename `FieldSchema.field` to `Fieldschema.field_info`
candleindark Jun 9, 2024
5525844
Add `field_name` to `FieldSchema`
candleindark Jun 9, 2024
ac1a904
Remove `field_name` param in `SlotGenerate.__init__()`
candleindark Jun 9, 2024
2460cb4
Implement `SlotGenerator._definitions_schema()`
candleindark Jun 9, 2024
6919703
Implement `SlotGenerator._definition_ref_schema()`
candleindark Jun 9, 2024
2b9209c
Reformat docstrings
candleindark Jun 9, 2024
934af9c
Implement translation of `core_schema.NoneSchema`
candleindark Jul 4, 2024
9709446
Eliminate unneeded variable with inlining
candleindark Jul 4, 2024
ae63ced
Implement translation of `core_schema.BoolSchema`
candleindark Jul 5, 2024
d737297
Implement translation of `core_schema.IntSchema`
candleindark Jul 6, 2024
44fc668
Define and utilize `SlotGenerator._attach_note` helper method
candleindark Jul 8, 2024
f913760
Define `has_exactly_one_truthy` for use in tests
candleindark Jul 8, 2024
1a4378d
Implement the transaltion of `core_schema.FloatSchema`
candleindark Jul 8, 2024
c2f45e7
Improve docstrings
candleindark Jul 8, 2024
6e4e858
Implement translation of `core_schema.DecimalSchema`
candleindark Jul 9, 2024
569537a
Implement translation of `core_schema.StringSchema`
candleindark Jul 11, 2024
670808e
RF: Introduce `in_exactly_one_string()` and `in_no_string()` helper f…
candleindark Jul 11, 2024
3fdc8ff
RF: Introduce `verify_str_lst()` helper func
candleindark Jul 12, 2024
71b212f
Implement translation of `core_schema.DateSchema`
candleindark Jul 12, 2024
50d9b95
Implement translation of `core_schema.TimeSchema`
candleindark Jul 13, 2024
bb3acb8
Implement translation of `core_schema.DatetimeSchema`
candleindark Jul 13, 2024
3dfc852
Improve notes of generate slot
candleindark Jul 13, 2024
613759d
Initialize meta slot `required` to True
candleindark Jul 14, 2024
c80da13
Move any setup of the schema to `LinkmlGenerator.generate()`
candleindark Jul 14, 2024
24b73e4
Implement translation of `core_schema.LiteralSchema`
candleindark Jul 15, 2024
249db58
Implement translation of `core_schema.EnumSchema`
candleindark Jul 15, 2024
c66ded3
Move some of the imports outside of tests
candleindark Jul 15, 2024
f298513
Improve docstrings
candleindark Jul 15, 2024
907e4bf
Implement translation of `core_schema.ListSchema`
candleindark Jul 16, 2024
c27140d
Update note
candleindark Jul 17, 2024
a5e3d5e
Complete translation of all function schemas
candleindark Jul 17, 2024
da2935e
Complete translation of `core_schema.WithDefaultSchema`
candleindark Jul 20, 2024
a0b147c
Complete translation of `core_schema.NullableSchema`
candleindark Jul 21, 2024
c439830
Provide a temp translation of union types
candleindark Jul 21, 2024
3e27151
Complete translation of `core_schema.ModelSchema` to slot
candleindark Jul 22, 2024
33095db
Complete translation of `core_schema.UrlSchema`
candleindark Jul 22, 2024
ac99791
rename into generate_linkml_from_pydantic.py since we make it generic
yarikoptic Jul 23, 2024
28a53f3
RF: make generate_linkml_from_pydantic.py use new code
yarikoptic Jul 23, 2024
c1b4c0a
Add pyndatic explicitly to requirements.txt
yarikoptic Jul 23, 2024
e154af1
Add tests github CI workflow (untested yet, based on simple one in ci…
yarikoptic Jul 23, 2024
0f626c2
RF: delay typer.app application to the __main__ time, so our main sta…
yarikoptic Jul 23, 2024
a8bf3c8
RF on how we load modules and submodules -- now we would not mask awa…
yarikoptic Jul 23, 2024
68281ec
Print out some informative messages from CLI
yarikoptic Jul 23, 2024
330758a
Update test class name
candleindark Jul 23, 2024
f85cf6b
Improve comments in `get_all_modules`
candleindark Jul 24, 2024
1cde0e3
Remove unused variable
candleindark Jul 24, 2024
c88d9b6
Remove old implementation from `get_all_modules()`
candleindark Jul 24, 2024
415d071
Have `get_all_modules()` return a set
candleindark Jul 24, 2024
c6303ee
Update `test_smoke_cli()`
candleindark Jul 24, 2024
0c8e0e7
Update `test_instantiation_with_definitions_in_module`
candleindark Jul 24, 2024
3943af0
Make `get_all_modules()` more flexible
candleindark Jul 24, 2024
75bc895
Improve the docstring of `get_all_modules()`
candleindark Jul 24, 2024
a1aeff5
Update `test_smoke()`
candleindark Jul 24, 2024
62902be
Mark `test_smoke()` as `xfail`
candleindark Jul 24, 2024
3a310c6
Mark `test_smoke_cli()` as `xfail`
candleindark Jul 24, 2024
980fa44
Update `test_smoke_cli()`
candleindark Jul 24, 2024
011d2af
Add ` pytest-xdist` as a dependency
candleindark Jul 24, 2024
c359580
Reformat file
candleindark Jul 24, 2024
5956209
Remove 3.8 from the versions of Python to be tested with in workflow
candleindark Jul 24, 2024
0121044
Run pytest with `--numprocesses=logical` in workflow
candleindark Jul 24, 2024
cb5db65
Implement `get_uuid_regex()`
candleindark Jul 26, 2024
70df4b7
Move import statement outside of tests in `test_tools.py`
candleindark Jul 26, 2024
50b54ae
Default `version` param of ``get_uuid_regex()` to `None`
candleindark Jul 26, 2024
1fa7967
Complete translation of `core_schema.UuidSchema`
candleindark Jul 26, 2024
0e631da
Improve docstrings
candleindark Jul 26, 2024
ec65faf
Define `TranslationNotImplementedError`
candleindark Jul 26, 2024
cacc614
Replace use of `NotImplementedError` with `TranslationNotImplementedE…
candleindark Jul 26, 2024
754f5c1
Define and utilize `models_and_enums()`
candleindark Jul 26, 2024
0037293
Define `linkml_generator()` fixture
candleindark Jul 26, 2024
45a9def
Mark expected failure test case for `test_instantiation_with_definiti…
candleindark Jul 27, 2024
fe021f9
RF: Simplify LinkmlGenerator creation from `models_and_enums` fixture
candleindark Jul 27, 2024
ca01c2b
Add todo for adding multiple module names test cases
candleindark Jul 27, 2024
d05a9ee
Remove `NotImplementedError` from `LinkmlGenerator._add_classes()`
candleindark Jul 27, 2024
50057b6
Implement `TestLinkmlGenerator.test_generate()`
candleindark Jul 27, 2024
0acf06f
Move casting of `context` in `resolve_ref_schema()`
candleindark Jul 28, 2024
6b0cfe5
Improve comment
candleindark Jul 28, 2024
f4c4d8f
RF: Introduce variable
candleindark Jul 28, 2024
6f25bf9
Introduce and utilize `model` attribute in `FieldSchema`
candleindark Jul 28, 2024
6e7493f
Modify message of `TranslationNotImplementedError`
candleindark Jul 28, 2024
db97cfe
Implement translation of `core_schema.JsonOrPythonSchema`
candleindark Jul 29, 2024
83f38e1
Adjust the formating of warning messages
candleindark Jul 29, 2024
f35a596
Complete translation of `core_schema.ChainSchema`
candleindark Jul 29, 2024
d10a07d
Complete temp translation of `core_schema.DictSchema`
candleindark Jul 29, 2024
2700e87
Remove unneeded `xfail` pytest marks
candleindark Jul 29, 2024
b3a1aa1
Add initial setup by Hatch
candleindark Aug 5, 2024
a7c62f9
Remove generated LinkML models
candleindark Aug 5, 2024
41bbf7f
Adjust pyproject.toml generated by Hatch to this project
candleindark Aug 5, 2024
3de9b6c
Specify the test environment
candleindark Aug 5, 2024
f933856
Move `pydantic2linkml` package into the `src` folder
candleindark Aug 5, 2024
fe8517e
Create `__about__.py`
candleindark Aug 5, 2024
05fcf19
Remove `__init__.py` in the project directory
candleindark Aug 5, 2024
31c84af
Add `dandischema` and `aind-data-schema` dependencies for test env
candleindark Aug 5, 2024
feb22c0
Reorganize generate_linkml_from_pydantic.py into CLI of the package
candleindark Aug 5, 2024
95c6fe4
Make tests a package
candleindark Aug 6, 2024
eed63c1
Move test files out of test_pydantic2linkml pacakge
candleindark Aug 6, 2024
e646ccf
Move and rename test from test_converters.py to test_cli.py
candleindark Aug 6, 2024
911398e
Remove requirements.txt
candleindark Aug 6, 2024
0aae765
Update GitHub workflow run tests using Hatch
candleindark Aug 6, 2024
f9e5c09
Remove Python spec in `test` and `types` envs
candleindark Aug 6, 2024
4beac0c
Format and lint source code using `hatch fmt`
candleindark Aug 7, 2024
01d24cc
Allow log level be specified through CLI
candleindark Aug 11, 2024
9ec3c0b
Reformat file
candleindark Aug 11, 2024
63a53af
Log execution status messages to `stderr`
candleindark Aug 11, 2024
22e98f3
Replace use of `warnings.warn` with `Logger.warning`
candleindark Aug 11, 2024
d550b8d
Correct type annotation of `output_file` in `cli.main()`
candleindark Aug 11, 2024
3f3f246
Add short names to CLI options
candleindark Aug 12, 2024
d0f3de9
Remove the use of `_generate_next_value_()` in `Enum` subclass
candleindark Aug 12, 2024
45fb741
Lower default env to the lowest supported Python version
candleindark Aug 12, 2024
7103f4e
Refactor out code into `gen_linkml.translate_defs()`
candleindark Aug 12, 2024
12bf5d2
Deduplicate input module names in `tools.get_all_modules()`
candleindark Aug 12, 2024
eae46d6
Deduplicate input module names in `gen_linkml.translate_defs()`
candleindark Aug 12, 2024
923bb22
Annotate arg in `gen_linkml.translate_defs()`
candleindark Aug 12, 2024
ef0605e
Improve docs of `gen_linkml.translate_defs()`
candleindark Aug 12, 2024
4856745
Implement and utilize `tools.force_to_set()`
candleindark Aug 18, 2024
65135d2
Type annotate return types in `LinkmlGenerator`
candleindark Aug 19, 2024
adef51f
Define and use the interface of `LinkmlGenerator._add_slot()`
candleindark Aug 19, 2024
b41d8ca
Implement `LinkmlGenerator._add_slot()`
candleindark Aug 20, 2024
37277b7
Sort models and enums before they are translated
candleindark Aug 20, 2024
199aadf
Implement `tools.sort_dict`
candleindark Aug 25, 2024
515abef
Ensure slot definitions are sorted
candleindark Aug 25, 2024
acbf1cb
Rearrange code
candleindark Aug 26, 2024
2f587fe
Specify the interface for adding classes
candleindark Aug 27, 2024
ca92266
Let the first argument to `pytest.mark.parametrize` be a tuple
candleindark Aug 27, 2024
a33c595
Use absolute imports instead of relative
candleindark Aug 27, 2024
3f618c5
Merge branch 'enh' into add-classes
candleindark Aug 27, 2024
f4a59a2
Introduce `GeneratorReuseError`
candleindark Aug 27, 2024
4dbf8ba
Define exception message in `TranslationNotImplementedError`
candleindark Aug 27, 2024
c9923c8
Merge branch 'enh' into add-classes
candleindark Aug 27, 2024
717205b
Outline class generation process
candleindark Aug 27, 2024
a2b1169
Define slots in `ClassDefinition` objects
candleindark Aug 27, 2024
6975a5a
Eliminate use of `from __future__ import annotations` in tools.py
candleindark Aug 28, 2024
ec092b7
Eliminate use of `from __future__ import annotations` in test_tools.py
candleindark Aug 28, 2024
b36e30a
Merge branch 'enh' into add-classes
candleindark Aug 28, 2024
46ef009
Simply the key function for sorting slots by names
candleindark Aug 28, 2024
134e685
Remove unneeded comment
candleindark Aug 29, 2024
73f5815
Define `SlotExtensionError`
candleindark Aug 30, 2024
b5155fd
Define `get_non_empty_meta_slots()`
candleindark Sep 2, 2024
647ebda
Raise `ValueError` in `SlotExtensionError` creation
candleindark Sep 2, 2024
3af3e4e
Define `get_slot_usage_entry`
candleindark Sep 2, 2024
e736339
RF: Introduce `sort_dict_by_ikeys()`
candleindark Sep 3, 2024
3cab658
Introduce a dict for slot representation of new fields
candleindark Sep 3, 2024
cfc7261
Add slot usage entries for new slots
candleindark Sep 3, 2024
160a7d4
Ensure slot usage entries are sorted
candleindark Sep 3, 2024
843d145
Delay sorting of slots
candleindark Sep 3, 2024
959b5e3
Rename variable
candleindark Sep 3, 2024
1e4a16b
Improve docstring of `SlotExtensionError`
candleindark Sep 4, 2024
cf45afb
Handling overriding fields in class generation
candleindark Sep 4, 2024
4a6ab7c
Set parent class of a class definition
candleindark Sep 4, 2024
dd50a66
Rename `test_get_parent_model` to `test_get_parent_models`
candleindark Sep 4, 2024
c003aea
Make `pydantic.BaseModel` an invalid input to `get_parent_models`
candleindark Sep 5, 2024
0810e8d
Specify mixins in generating a `ClassDefinition`
candleindark Sep 5, 2024
f839ad8
Specify mixins in generating a `ClassDefinition`
candleindark Sep 5, 2024
3ec2098
Add the interface for translating complex number specs
candleindark Sep 6, 2024
f8e1491
Restrict pydantic to ~=2.9
candleindark Sep 6, 2024
8bab6d1
Update `.gitignore` to ignore `__pycache__` folders
candleindark Sep 12, 2024
640f9c6
Update README.md
candleindark Sep 12, 2024
27ec216
Update project urls
candleindark Sep 13, 2024
9c33b17
fix: Make `SlotGenerator._complex_schema()` contingent on Pydantic ve…
candleindark Oct 15, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
name: Test

on:
pull_request:
branches: [ main ]
push:
branches: [ main ]
tags:
- '*'
workflow_dispatch:

jobs:
test:
runs-on: ${{ matrix.runs-on }}
strategy:
fail-fast: false
matrix:
runs-on:
- macos-latest
- windows-latest
- ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4

- name: Install Hatch
uses: pypa/hatch@install

- name: Run tests
run: hatch run test:python -m pytest --numprocesses=logical -s -v tests
env:
# Needed for coveralls:
GITHUB_TOKEN: ${{ secrets.COVERALLS_REPO_TOKEN }}

# vim:set et sts=2:
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1 +1,4 @@
.idea/

# === Python ===
__pycache__/
14 changes: 8 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,12 @@
# linkml-model-generator
Translate pydantic-based schemas from python modules to LinkML.
# pydantic2linkml
A tool for translating models expressed in Pydantic to LinkML

### Run
[![Hatch project](https://img.shields.io/badge/%F0%9F%A5%9A-Hatch-4051b5.svg)](https://github.com/pypa/hatch)

To run the program, use:
-----

```
python3 generate_linkml_from_aind.py --root_module_name="aind_data_schema.models" --output_file="aind.yml"
### Sample Run

```console
pydantic2linkml -o o.yml -l INFO dandischema.models
```
90 changes: 0 additions & 90 deletions generate_linkml_from_aind.py

This file was deleted.

Loading