You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I've been playing around with EventListeners trying to automatically generate lineage for my queries. I'm a huge fan of how the sources of each output column are mapped on a per column basis in the ioMetadata making this operation very easy.
Something I've noticed however, is that only column projections and not predicates are present in the resulting metadata sources.
INSERT INTOmemory.default.output
SELECT*FROMmemory.default.input
WHERE input_value >= (SELECTmax(threshold) frommemory.default.threshold);
In the resulting sources for the target column output_value, we only get column input_value from memory.default.input. Is there a better way to determine this at run time than an EventListener? I would like to see memory.default.threshold's threshold column as a source, preferably with a way to differentiate between predicate and projection lineage.
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hey there!
I've been playing around with EventListeners trying to automatically generate lineage for my queries. I'm a huge fan of how the sources of each output column are mapped on a per column basis in the ioMetadata making this operation very easy.
Something I've noticed however, is that only column projections and not predicates are present in the resulting metadata sources.
An example:
DDL
DML
In the resulting sources for the target column output_value, we only get column input_value from memory.default.input. Is there a better way to determine this at run time than an EventListener? I would like to see memory.default.threshold's threshold column as a source, preferably with a way to differentiate between predicate and projection lineage.
Beta Was this translation helpful? Give feedback.
All reactions