Structs, Arrays, Repeated, and FQNs

[amiorin]

Use case

As a Data steward, I'd like to tag and add relationships like upstream to fields of repeated structs. These kind of tables are very common in big data because they denormalize two tables with a "has many" relationship in a single table because of performance reasons. Nevertheless governance and discoverability is still a requirement for them and OpenMetadata should support them like it supports traditional flatten or nested without repeated tables.

BigQuery defines a new type called repeated that it is equivalent to array<struct>>. https://medium.com/google-cloud/bigquery-explained-working-with-joins-nested-repeated-data-1941646ccb5b

Solution

Here we advocate for the adoption of repeated in OpenMetadata.
Advantages:

• FQN are simpler, just a dot: user.sessions.clicks.ip, User has many sessions. Session has many clicks. A click has an IP that it is tagged as PII.

Implementation details

• It should be a flag in the Python code to enable the conversion of array<struct> to repeated.

References

• Discussion in Slack: https://openmetadata.slack.com/archives/C02DZK3QH8Q/p1648659215556649
• Issue in GitHub: Parse and Expand Arrays of Structs #3763