Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Output entity hints for dynamic tasks #6

Draft
wants to merge 33 commits into
base: master
Choose a base branch
from
Draft

Output entity hints for dynamic tasks #6

wants to merge 33 commits into from

Conversation

Tom-Newton
Copy link
Owner

TL;DR

Please replace this text with a description of what this PR accomplishes.

Type

  • Bug Fix
  • Feature
  • Plugin

Are all requirements met?

  • Code completed
  • Smoke tested
  • Unit tests added
  • Code documentation added
  • Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

https://github.com/flyteorg/flyte/issues/

Follow-up issue

NA
OR
https://github.com/flyteorg/flyte/issues/

@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch 2 times, most recently from 5fe0ef7 to 45b9b19 Compare December 1, 2023 20:21
@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch 9 times, most recently from 8f8e46d to 920a5d2 Compare December 19, 2023 20:52
@Tom-Newton
Add output_entity_hint and support for serialising it
3c90b27 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Correct assertion
c18f69e 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Add output_entity_hint to task decorator
26fc9b2 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Type hints
3c5b696 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Fix circular import of type hints
cc00802 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Support iterable of output_entity_hints (#5)
e37e96d 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Fix circular import for type hints
bcc90f0 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Fix tests
e18510c 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Auto-format
4672cfc 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Add positive test case
2be6e69 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Add a test for disallowing entity hints on static tasks
7f7ab2a 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
raise instead of assert
89ef3e9 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Rename output_entity_hints -> node_dependency_hints
a1d1016 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Update docstrings
fb97a71 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Auto-format
6cf0205 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Fix test_serialize_vscode
cf1cba5 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton
Maybe fix the docs error
8d1e734 
Signed-off-by: Thomas Newton <[email protected]>
@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch from 920a5d2 to a29cd98 Compare December 20, 2023 00:15
@Tom-Newton Tom-Newton force-pushed the tomnewton/upsream_output_entity_hints branch from a29cd98 to 62c5209 Compare December 20, 2023 00:15
Tom-Newton and others added 15 commits December 20, 2023 15:41
@Tom-Newton
Fix monodocs build
6ae0e89 
Signed-off-by: Thomas Newton <[email protected]>
@eapolinario
Use logger in data_persistence.py (flyteorg#2129)
cead901 
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@neilisaur
Adjust tar method to iterate over files/dirs in dir rather than strip… (
956e0be 
flyteorg#2131)

Signed-off-by: Neil Stout <[email protected]>
@Tom-Newton
Include exception type in error messages (flyteorg#2130)
de0644f 
* Add test

Signed-off-by: Thomas Newton <[email protected]>

* Fix

Signed-off-by: Thomas Newton <[email protected]>

* Better formatting

Signed-off-by: Thomas Newton <[email protected]>

---------

Signed-off-by: Thomas Newton <[email protected]>
@thomasjpfan
Adds get_default_image into configuration plugin (flyteorg#2133)
859b0eb 
Signed-off-by: Thomas J. Fan <[email protected]>
@honnix
Add 3.12 as classifier (flyteorg#2135)
a4975b6 
Signed-off-by: Hongxin Liang <[email protected]>
@neilisaur
Fixing copy-all version of tar file creation as well (flyteorg#2134)
f058663 
Signed-off-by: Neil Stout <[email protected]>
@dependabot
Bump pillow from 10.1.0 to 10.2.0 in /plugins/flytekit-onnx-pytorch (f…
f9eaf55 
…lyteorg#2127)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.1.0 to 10.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.1.0...10.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp from 3.8.6 to 3.9.2 (flyteorg#2137)
d7dfbaf 
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.8.6 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.8.6...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp from 3.9.1 to 3.9.2 in /plugins/flytekit-spark (flyteorg…
b763758 
…#2140)

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.1 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.9.1...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp from 3.9.0 to 3.9.2 in /plugins/flytekit-airflow (flyteo…
d10bda1 
…rg#2139)

Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.0 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.9.0...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump aiohttp (flyteorg#2138)
1b8ad7e 
Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.1 to 3.9.2.
- [Release notes](https://github.com/aio-libs/aiohttp/releases)
- [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst)
- [Commits](aio-libs/aiohttp@v3.9.1...v3.9.2)

---
updated-dependencies:
- dependency-name: aiohttp
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@dependabot
Bump pillow from 10.1.0 to 10.2.0 in /plugins/flytekit-onnx-tensorflow (
1789db9 
flyteorg#2126)

Bumps [pillow](https://github.com/python-pillow/Pillow) from 10.1.0 to 10.2.0.
- [Release notes](https://github.com/python-pillow/Pillow/releases)
- [Changelog](https://github.com/python-pillow/Pillow/blob/main/CHANGES.rst)
- [Commits](python-pillow/Pillow@10.1.0...10.2.0)

---
updated-dependencies:
- dependency-name: pillow
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
@eapolinario
Envvars local execution (flyteorg#2132)
8861d0a 
* Pass environment variables to local executions

Signed-off-by: Eduardo Apolinario <[email protected]>

* Pass environment variables to local executions

Signed-off-by: Eduardo Apolinario <[email protected]>

* Fix test_get_entities_in_file

Signed-off-by: Eduardo Apolinario <[email protected]>

* Lint

Signed-off-by: Eduardo Apolinario <[email protected]>

---------

Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
@eapolinario
Merge remote-tracking branch 'origin' into tomnewton/upsream_output_e…
5901cff 
…ntity_hints

Signed-off-by: Eduardo Apolinario <[email protected]>
Tom-Newton pushed a commit that referenced this pull request Apr 8, 2024
@wild-endeavor @eapolinario
art id proto (flyteorg#1928)
d2c6353 
* temp

Signed-off-by: Yee Hing Tong <[email protected]>

* stuff

Signed-off-by: Yee Hing Tong <[email protected]>

* temp

Signed-off-by: Yee Hing Tong <[email protected]>

* scaffolding areas mostly identified

Signed-off-by: Yee Hing Tong <[email protected]>

* add artifact to upload request

Signed-off-by: Yee Hing Tong <[email protected]>

* remove an unnecessary line in workflow

Signed-off-by: Yee Hing Tong <[email protected]>

* finish adding examples use cases maybe

Signed-off-by: Yee Hing Tong <[email protected]>

* add project/dom to get query

Signed-off-by: Yee Hing Tong <[email protected]>

* add from flyte idl

Signed-off-by: Yee Hing Tong <[email protected]>

* add project domain to as query

Signed-off-by: Yee Hing Tong <[email protected]>

* add condition in parameter to flyte idl

Signed-off-by: Yee Hing Tong <[email protected]>

* test stuff

* Remove artifactID from literal oneof, add to metadata (#2)

* Triggers (#6)

* Minor changes to get time series example working flyteorg#8

Signed-off-by: Yee Hing Tong <[email protected]>

* switch channel (flyteorg#10)

Signed-off-by: Yee Hing Tong <[email protected]>

* fix tests ignore - pr into other pr (flyteorg#1858)

Signed-off-by: Yee Hing Tong <[email protected]>

* Artf/update idl ux (flyteorg#1920)


Signed-off-by: Yee Hing Tong <[email protected]>

* Artf/trigger (flyteorg#1948)

* Add triggers
* Remove bind_partition_time and just assume users won't use that. It's just time_partition in the normal call function now.

Signed-off-by: Yee Hing Tong <[email protected]>

* remove the now deleted artifact spec (flyteorg#1984)

Signed-off-by: Yee Hing Tong <[email protected]>

* Literal metadata model update (flyteorg#2089)


Signed-off-by: Yee Hing Tong <[email protected]>

* Separate time partition (flyteorg#2114)



Signed-off-by: Yee Hing Tong <[email protected]>

* Split service code (flyteorg#2136)


Signed-off-by: Yee Hing Tong <[email protected]>

* remove empty files

Signed-off-by: Yee Hing Tong <[email protected]>

* add noneness check to metadata and add test

Signed-off-by: Yee Hing Tong <[email protected]>

* remove sandbox test for now

Signed-off-by: Yee Hing Tong <[email protected]>

* Artf/cleanup (flyteorg#2158)

* add a test

Signed-off-by: Yee Hing Tong <[email protected]>

* try updates

Signed-off-by: Yee Hing Tong <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>

* Use python 3.9 to run make doc-requirements.txt

Signed-off-by: Eduardo Apolinario <[email protected]>

* reasons not msg

Signed-off-by: Yee Hing Tong <[email protected]>

---------

Signed-off-by: Yee Hing Tong <[email protected]>
Signed-off-by: Eduardo Apolinario <[email protected]>
Co-authored-by: Eduardo Apolinario <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Tom-Newton @eapolinario @neilisaur @thomasjpfan @honnix