[Improve][Core Moduel]Support source can use dynamic column definetion.

[hk-lrzy]

Description

Motivation

Now in the bitsail, the column is an necessary option and must appear in the job.reader configuration, in most of case it could be work and make sense.

(The screenshot show the logic of bitsail about how to parse RowTypeInfo from column option)

At same time CDC become more and more popular and bitsail also want to support it in recently months. In CDC situation, the column may not need to declare in the configuration at first, cause it's not fixed and even will change in the runtime.

Here we have two option to solve the issue:

• Option 1: column definetion not just from configuration file, but also from source&source reader. Like we will create new interface to show the prefer TypeInfo:

public interface PreferTypeInfo {

  RowTypeInfo getPreferTypeInfo();
}

And

    if (source instanceof PreferTypeInfo) {
      this.rowTypeInfo = ((PreferTypeInfo) source).getPreferTypeInfo()
    } else {
      List<ColumnInfo> columnInfos = readerConfiguration
          .get(ReaderOptions.BaseReaderOptions.COLUMNS);
      this.rowTypeInfo = TypeInfoUtils
          .getRowTypeInfo(source.createTypeInfoConverter(), columnInfos);
    }

Then cdc source could use custom typeinfo.

• Option 2: Create new option to declare we will be specify column definetion, like we create new option which named row-type and value is cdc to show there will be use cdcRowTypeInfo, and legacy value means we will use column option to generate the type info.

I perfer the Option 1

BitSail Component or Code Module

BitSail Core

Are you willing to submit PR?

• Yes, I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct