EntityReference and Lineage

[pmbrull]

Hi @sureshms, @harshach,

Thanks again for pushing this project, having a lot of fun with this. Only the theory-crafting part is already interesting to do :D

One question that popped into my mind is related to the integrations of the EntityReference and Lineage, as per my view on this it seems that their meaning might overlap in some situations.

For example, having a DashboardService as a source for multiple Dashboards whom at the same time aggregate different Charts: All these relationships are EntityReferences as per our definition of the Entities, where we require to have these ingredients for the Entity to be complete.

A different scenario would be a Pipeline. With the Lineage we will be able to specify that it has a series of Sources and Sinks, which will be at the same time other Table Entities (for simplicity).

What would be the relationship definition of:

• An Entity A has a Reference relationship with an Entity B vs.
• An Entity A is only the Source/Sink of an Entity B.

How is the relationship Pipeline vs. Table different than Dashboard vs. Chart? Or Model vs. Chart? I think it would be valuable to come up with a closed definition to help us evolve the Entities in a sustainable way.

Aside from the philosophical discussion, IMO it would make sense to update how the API works with the EntityReference to automatically draw the Lineage of those entities. This way it would be easier to have an auto-generated knowledge graph of our assets for a big chunk of the already present Entities.

Happy to discuss! 🌻

Thanks!

[sureshms]

@pmbrull, when we talk about metadata graph, the relationships go beyond lineage/provenance. There are other kinds of relationships such as a dashboard contains a chart, a user belongs to a team along with the lineage relationship, a data asset is upstream of another data asset. These relationships are captured in catalog-rest-service/src/main/java/org/openmetadata/catalog/jdbi3/Relationship.java. You are right, we should clean up these relationships at some point in time to reduce similar and overlapping definitions.

EntityReference is just a pointer to another entity. Such a pointer is used for capturing all relationships (1:1, 1:n) in the schemas. One such relationship is lineage.

How is the relationship Pipeline vs. Table different than Dashboard vs. Chart? Or Model vs. Chart? I think it would be valuable to come up with a closed definition to help us evolve the Entities in a sustainable way.
As I said earlier, there are lineage relationships (upstream) vs. other relationships - a service contains databases, a database contains tables, etc.

Aside from the philosophical discussion, IMO it would make sense to update how the API works with the EntityReference to automatically draw the Lineage of those entities. This way it would be easier to have an auto-generated knowledge graph of our assets for a big chunk of the already present Entities.
+1 for this. All entities have clearly defined relationships (as entity fields). A knowledge graph (or metadata graph) is a key goal of this project. We also should be able to have an interesting use of this to go beyond the current lineage/provinence user experience and expand to show a bigger picture of data.

Additionally, I think we should look into where stronger relationship an entity contains another (database contains tables, and a table belongs to single database) and weaker relationships a dashboard refers to a chart (because multiple dashboards can refer to the same chart) or a user is a member of a team (and user can be a member of many different teams).

[pmbrull]

Thanks, @sureshms for pointing me to the class! Really helpful to get all the possible relationships in writing.

I get your point about strong vs. weak relationships. Should we then have a new enum to differentiate those? I could imagine something like:

• Contains representing a weak relation, such as a Dashboard contains a Chart, where the Chart information is complete by itself and the Dashboard is just an aggregator/container.
• Belongs for strong relationships, e.g. a Table belongs to a Database, as the Table is not complete without its db information. Or a Dashboard belongs to a DashboardService.

IDK if the differentiation in terms of information completeness of the Entity and Entities acting as logical aggregators would still be too subjective.