You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I have a node which returns dict[str, Callable] for kedro to save my partitioned data. I've often had cases where it was failing mid way due to edge case i didn't cover and execution starts from all over again.
Context
I would need this to speed up experimentation in kedro and reduce unnecessary costs which may occur by re-running the node.
Possible Implementation
Adding a new parameter to PartitionDataset to support skipping already existing files. Something like use_cache: True
Possible Alternatives
I can def inherit the class and implement this but i thought it would be useful feature to have it in the core code.
The text was updated successfully, but these errors were encountered:
Description
I have a node which returns
dict[str, Callable]
for kedro to save my partitioned data. I've often had cases where it was failing mid way due to edge case i didn't cover and execution starts from all over again.Context
I would need this to speed up experimentation in kedro and reduce unnecessary costs which may occur by re-running the node.
Possible Implementation
Adding a new parameter to
PartitionDataset
to support skipping already existing files. Something likeuse_cache: True
Possible Alternatives
I can def inherit the class and implement this but i thought it would be useful feature to have it in the core code.
The text was updated successfully, but these errors were encountered: