From 33623f3717047b8099db951217180cb3c26f295f Mon Sep 17 00:00:00 2001 From: BInwei Yang Date: Fri, 24 Jan 2025 23:25:48 -0800 Subject: [PATCH 1/2] Update Configuration.md --- docs/Configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Configuration.md b/docs/Configuration.md index e217be45ff80..6647aa958979 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -21,7 +21,7 @@ You can add these configurations into spark-defaults.conf to enable or disable t | spark.sql.join.preferSortMergeJoin | When true, prefer sort merge join over shuffled hash join.
Note: Please turn off preferSortMergeJoin. | false | | spark.plugins | To load Gluten's components by Spark's plug-in loader | org.apache.gluten.GlutenPlugin | | spark.shuffle.manager | To turn on Gluten Columnar Shuffle Plugin | org.apache.spark.shuffle.sort.ColumnarShuffleManager | -| spark.gluten.enabled | Enable Gluten, default is true. Just an experimental property. Recommend to enable/disable Gluten through the setting for `spark.plugins`. | true | +| spark.gluten.enabled | Enable Gluten at runtime, default is true. It fallbacks to vanilla Spark for all query plans. Recommend to enable/disable Gluten through the setting for `spark.plugins`. | true | | spark.gluten.memory.isolation | (Experimental) Enable isolated memory mode. If true, Gluten controls the maximum off-heap memory can be used by each task to X, X = executor memory / max task slots. It's recommended to set true if Gluten serves concurrent queries within a single session, since not all memory Gluten allocated is guaranteed to be spillable. In the case, the feature should be enabled to avoid OOM. Note when true, setting spark.memory.storageFraction to a lower value is suggested since storage memory is considered non-usable by Gluten. | false | | spark.gluten.ras.enabled | Enables RAS (relation algebra selector) during physical planning to generate more efficient query plan. Note, this feature doesn't bring performance profits by default. Try exploring option `spark.gluten.ras.costModel` for advanced usage. | false | | spark.gluten.sql.columnar.maxBatchSize | Number of rows to be processed in each batch. Default value is 4096. | 4096 | From 186ac215769fcfa433884099fd4eae093281eb37 Mon Sep 17 00:00:00 2001 From: BInwei Yang Date: Wed, 29 Jan 2025 12:25:18 -0800 Subject: [PATCH 2/2] Update Configuration.md --- docs/Configuration.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/Configuration.md b/docs/Configuration.md index 6647aa958979..d577f9de24e1 100644 --- a/docs/Configuration.md +++ b/docs/Configuration.md @@ -21,7 +21,7 @@ You can add these configurations into spark-defaults.conf to enable or disable t | spark.sql.join.preferSortMergeJoin | When true, prefer sort merge join over shuffled hash join.
Note: Please turn off preferSortMergeJoin. | false | | spark.plugins | To load Gluten's components by Spark's plug-in loader | org.apache.gluten.GlutenPlugin | | spark.shuffle.manager | To turn on Gluten Columnar Shuffle Plugin | org.apache.spark.shuffle.sort.ColumnarShuffleManager | -| spark.gluten.enabled | Enable Gluten at runtime, default is true. It fallbacks to vanilla Spark for all query plans. Recommend to enable/disable Gluten through the setting for `spark.plugins`. | true | +| spark.gluten.enabled | Enable Gluten at runtime, default is true. It fallbacks to vanilla Spark for all query plans if set to false. Recommend to enable/disable Gluten through the setting for `spark.plugins`. | true | | spark.gluten.memory.isolation | (Experimental) Enable isolated memory mode. If true, Gluten controls the maximum off-heap memory can be used by each task to X, X = executor memory / max task slots. It's recommended to set true if Gluten serves concurrent queries within a single session, since not all memory Gluten allocated is guaranteed to be spillable. In the case, the feature should be enabled to avoid OOM. Note when true, setting spark.memory.storageFraction to a lower value is suggested since storage memory is considered non-usable by Gluten. | false | | spark.gluten.ras.enabled | Enables RAS (relation algebra selector) during physical planning to generate more efficient query plan. Note, this feature doesn't bring performance profits by default. Try exploring option `spark.gluten.ras.costModel` for advanced usage. | false | | spark.gluten.sql.columnar.maxBatchSize | Number of rows to be processed in each batch. Default value is 4096. | 4096 |