Usage of Gluten Functions At Query Level

[rajatma1993]

Backend

VL (Velox)

Bug description

Hello Team,

I am currently testing Gluten to benchmark its performance against TPCH. In the process, I observed that in TPCH Query 16, In Flame Graph we see only 4.69% of the function usage of the overall seems to be utilized.

I would like to understand if this utilization level aligns with our expectations. Additionally, could we confirm if this level of utilization contributes significantly to the execution speedup?

Attached is the Flame Graph we have generated.

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[FelixYBW]

It depends on the partition data size of the stage. You may increase the TPCH SF or decrease the partition number to get larger partition in each task. The larger partition a task is processing the more time spending in native.