--conf spark.shuffle.manager=org.apache.spark.shuffle.sort.ColumnarShuffleManager causes failure

[sumokiama]

We are unable to use the ColumnarShuffleManager in any of the TPC* tests including the basic TPCH in-memory tests.
The query appears to execute but fails near the end with:

java.lang.RuntimeException: Error during calling Java code from native code: java.lang.RuntimeException: Failed to read message from shuffle.

Attached are the shell script and spark-shell output from a TPCDS/Q3/3TB using Centos8/S3.
gluten_columnar_shuffle_failure.txt
tpcds_s3.sh.txt

[ted-jenks]

We ran into these errors when we were running on an unsupported version of Java. See here: https://github.com/oap-project/gluten/blob/main/docs/get-started/Velox.md#:~:text=Currently%2C%20java%208%20is%20required%20and%20the%20support%20for%20java%2011/17%20is%20not%20ready

We were able to get around this though with the following JVM flags:

--illegal-access=permit -Dio.netty.tryReflectionSetAccessible=true --add-opens java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.util=ALL-UNNAMED

--illegal-access=permit is an option from Java 9 to Java 16 which opens each package of platform modules to unnamed modules and shows a warning message only once. This allows native code to access package/module private stuff by reflection.
-Dio.netty.tryReflectionSetAccessible=true allows arrow to work on Java 9+ (already use this in foundry-spark).
--add-opens java.base/java.lang=ALL-UNNAMED allows everything in java.lang to be accessed by unnamed modules at runtime (including private stuff).
--add-opens=java.base/java.util=ALL-UNNAMED does the same as above for java.util.