How does OpenMetadata identify the changes and inconsistencies between the metadata and the database?

[ShilpaVernekar]

How is the metadata validation process carried out? How are the changes in the column of a table identified? Like a change in the data type, length, or a new column. How does OpenMetadata identify the changes and inconsistencies between the metadata and the database?

Originally answered in Slack support

[ShilpaVernekar]

The process will be as follows:

1. User will configure an Ingestion Connector to connect to MySQL or any other database.
2. This is a DAG that runs on Airflow at a scheduled interval to pull in metadata.
3. Lets say, there is a users table users(id, first_name, last_name, zipcode (BIGINT))
4. When the ingestion runs, it will retrieve that schema and ingest into OpenMetadata. On the first run, the zipcode is BIGINT
5. User then changes this schema in the MySQL database and changes the BIGINT to Varchar and now users(id, first_name, last_name, zipcode (VARCHAR))
6. Next time when the dag runs, it will pick up these changes and send them to OpenMetadata.

OpenMetadata does the validation against existing schema and recognizes this change is a backward incompatible and creates a major version change and updates the schema on OpenMetadata side. So one can look at the versions to see how the schema has evolved and what changed. The priority is always the last definition taken from the database, which is assumed to be the correct one or at least the most recent one.