Gluten 2025 Roadmap

[FelixYBW]

Description

2024 is almost gone. Time to define 2025 features. Here is some from my mind. Feel free to add more or comment
For more information about the achievements in 2024 Roadmap, please check #4709

Velox backend:

• Spark 4.0 support with minimal JDK 11 support, Deprecate Spark 3.2 and JDK 8
• Spark 3.3.x/3.4.x/3.5.x upgrade
• Switch to Upstream Velox's official release
• Stage level resource management(Stage level resource management to handle offheap/onheap memory conflict #4392 )
• Spill enhancement and performance ([VL] Spill related issues tracker #3030 )
• Hash table broadcast support in BHJ (https://docs.google.com/document/d/1upEby9aJnBcMKul7HS5agSr1zwFR63Iq4QL1nYxuFvU/edit?usp=sharing)
• Dictionary support in shuffle
• Multile core per task ([VL] Use multiple threads in the same executor #7810 )
• Pyspark Support including Python UDF/Arrow UDF
• Other Accelerator such as GPU, FPGA Support
• ARM architecture support
• Stability improvement, full fuzzer support, Result Mismatch issues ([VL] Result mismatch issues tracker #4652 )
• Spark Coverage including Operators and Functions
• Full Datalake support
• File Format support (Json)
• Performance improvement: Spill, Sort, HashAgg, HashJoin, ColumnarShuffle, and others
• Complex type in R2C ([VL] row2VeloxColumnar bad performance #7223)
• Gluten Indicator
• Gluten + Flink + Velox POC (experimental)
• ANSI support
• query trace

Community:

• Major Release: 1.4/1.5/1.6/1.7 in Q1/Q2/Q3/Q4 2025
• Minor Release: On Demand
• Apache TLP Submission in Q4 2025

[zhanglistar]

What about supporting Flink framework?

[Myracle]

Any plan or discussion about flink?

[zhztheplayer]

@zhanglistar @Myracle

It waits for discussion.

Does someone want to open / host a discussion for the topic? cc @apache/gluten-committers

[jinchengchenghh]

Do we need to do complex data type support and optimization such as row to column optimization for complex data type?

[FelixYBW]

Do we need to do complex data type support and optimization such as row to column optimization for complex data type?

Yes, currently R2C with complex type is too expensive

[MacChen01]

What about support openEuler OS ?

[FelixYBW]

What about support openEuler OS ?

Can vcpkg=on work on openEuler?

[felipepessoto]

@FelixYBW what you mean by Full Datalake support? Is it to improve support and performance for Delta tables?

[FelixYBW]

@FelixYBW what you mean by Full Datalake support? Is it to improve support and performance for Delta tables?

some features are not supported yet in Velox and Gluten.

[VaibhavFRI]

@FelixYBW what will be included in ARM architecture support? Will Gluten, Velox utilize SVE in ARM to improve performance?

[FelixYBW]

@FelixYBW what will be included in ARM architecture support? Will Gluten, Velox utilize SVE in ARM to improve performance?

It means we will setup CI or jenkins to verify on the ARM platform. SVE is part of the performance optimization work.

[surnaik]

@FelixYBW Happy new year and thanks for sharing the roadmap, will we add MOR table support for Hudi as part of full data lake support?

[zhouyuan]

@FelixYBW Happy new year and thanks for sharing the roadmap, will we add MOR table support for Hudi as part of full data lake support?

@surnaik Hi, yes there's a draft on this already, it's pending on Velox side change
#8056

[FelixYBW]

@surnaik Hi, yes there's a draft on this already, it's pending on Velox side change #8056

@zhouyuan do we have a list of missing support for delta lake like this?

[liujiayi771]

@zhouyuan @surnaik This PR does not support Hudi, it supports Iceberg's equality MOR table. We have already supported Iceberg's position MOR table. For Hudi, supporting MOR currently seems quite difficult. The native engine only has good support for deletion vector type MOR tables.

[vinothchandar]

cc @xushiyan @yihua.

[jinchengchenghh]

We can also support query trace.