[CH] sum with filter bad performance compared to vanilla spark

[taiyang-li]

Backend

CH (ClickHouse)

Bug description

gluten

0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = true; 
+-----------------------+--------+
|          key          | value  |
+-----------------------+--------+
| spark.gluten.enabled  | true   |
+-----------------------+--------+
1 row selected (0.034 seconds)
0: jdbc:hive2://localhost:10000/> 
0: jdbc:hive2://localhost:10000/> 
0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000);
+-----------------------------------+
| sum((IF(((id % 3) = 0), id, 0)))  |
+-----------------------------------+
| 1666666683333333                  |
+-----------------------------------+
1 row selected (64.729 seconds)
0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000);
+-----------------------------------+
| sum((IF(((id % 3) = 0), id, 0)))  |
+-----------------------------------+
| 1666666683333333                  |
+-----------------------------------+
1 row selected (64.811 seconds)

vanilla

0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = false; 
+-----------------------+--------+
|          key          | value  |
+-----------------------+--------+
| spark.gluten.enabled  | false  |
+-----------------------+--------+
1 row selected (0.09 seconds)
0: jdbc:hive2://localhost:10000/> select sum(id) filter(where id % 3 = 0) from range(100000000);
+----------------------------------------+
| sum(id) FILTER (WHERE ((id % 3) = 0))  |
+----------------------------------------+
| 1666666683333333                       |
+----------------------------------------+
1 row selected (0.333 seconds)

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[taiyang-li]

Update: after range operator is offloaded to CH. The performance of gluten is much faster, but still slower than vanilla spark (1.216s vs 0.333s).

0: jdbc:hive2://localhost:10000/> set spark.gluten.enabled = true; 
+-----------------------+--------+
|          key          | value  |
+-----------------------+--------+
| spark.gluten.enabled  | true   |
+-----------------------+--------+
1 row selected (0.045 seconds)
0: jdbc:hive2://localhost:10000/> select sum(if(id%3=0, id, 0)) from range(100000000); 
+-----------------------------------+
| sum((IF(((id % 3) = 0), id, 0)))  |
+-----------------------------------+
| 1666666683333333                  |
+-----------------------------------+
1 row selected (1.216 seconds)