[Umbrella][Feature] Support spark engine

[Alibaba-HZY]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

In roadmap,we plan to support the spark engine.
for kv table spark can support micro batch read, batch read, streaming writer and batch writer.
for log table spark can support micro batch read, streaming writer and batch writer.

According to @wuchong‘s suggestion,i will create some sub-tasks.

Task list

• spark catalog [Feature] Support spark catalog(base spark 3.3.3) #228
• streaming read @Alibaba-HZY
• batch read
• union read
• streaming write
• batch write

Solution

In task "streaming read" I'll implement it and introduce some base classes
such as
Introduce flussSparkTable implements org.apache.spark.sql.connector.catalog.Table
Introduce SparkCatalog implements org.apache.spark.sql.connector.catalog.TableCatalog
Introduce SparkInternalRow and SparkTypeUtils

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[XuQianJin-Stars]

Which versions of Spark are supported?

[wuchong]

Thanks @Alibaba-HZY for creating this issue! Could you create some sub-tasks that others can help together?

Regarding the priorities, because of Fluss is a streaming storage, IMO, the priorities can be streaming read > batch read > union read > streaming write > batch write.

Regarding the versions, let's start to support from Spark 3.5, and extend to Spark 3.4, 3.3 in the future.

[Alibaba-HZY]

Thanks @Alibaba-HZY for creating this issue! Could you create some sub-tasks that others can help together?

Regarding the priorities, because of Fluss is a streaming storage, IMO, the priorities can be streaming read > batch read > union read > streaming write > batch write.

Regarding the versions, let's start to support from Spark 3.5, and extend to Spark 3.4, 3.3 in the future.

ok done