Skip to content

Commit

Permalink
[SPARK-44719][SQL] Fix NoClassDefFoundError when using Hive UDF
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR changes jackson-mapper-asl's scope from `test` to `${hive.deps.scope}`.

### Why are the changes needed?

Fix `NoClassDefFoundError` when using Hive UDF:
```
spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar;
Time taken: 0.413 seconds
spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP';
Time taken: 0.038 seconds
spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10);
23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)]
java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory
	at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64)
	at java.lang.Class.forName0(Native Method)
	at java.lang.Class.forName(Class.java:348)
...
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

manual test.

Closes #42446 from wangyum/SPARK-44719.

Authored-by: Yuming Wang <[email protected]>
Signed-off-by: Kent Yao <[email protected]>
  • Loading branch information
wangyum authored and yaooqinn committed Aug 12, 2023
1 parent 567823e commit 7baf9da
Show file tree
Hide file tree
Showing 3 changed files with 9 additions and 10 deletions.
2 changes: 2 additions & 0 deletions dev/deps/spark-deps-hadoop-3-hive-2.3
Original file line number Diff line number Diff line change
Expand Up @@ -99,11 +99,13 @@ ini4j/0.5.4//ini4j-0.5.4.jar
istack-commons-runtime/3.0.8//istack-commons-runtime-3.0.8.jar
ivy/2.5.1//ivy-2.5.1.jar
jackson-annotations/2.15.2//jackson-annotations-2.15.2.jar
jackson-core-asl/1.9.13//jackson-core-asl-1.9.13.jar
jackson-core/2.15.2//jackson-core-2.15.2.jar
jackson-databind/2.15.2//jackson-databind-2.15.2.jar
jackson-dataformat-cbor/2.15.2//jackson-dataformat-cbor-2.15.2.jar
jackson-dataformat-yaml/2.15.2//jackson-dataformat-yaml-2.15.2.jar
jackson-datatype-jsr310/2.15.2//jackson-datatype-jsr310-2.15.2.jar
jackson-mapper-asl/1.9.13//jackson-mapper-asl-1.9.13.jar
jackson-module-scala_2.12/2.15.2//jackson-module-scala_2.12-2.15.2.jar
jakarta.annotation-api/1.3.5//jakarta.annotation-api-1.3.5.jar
jakarta.inject/2.6.1//jakarta.inject-2.6.1.jar
Expand Down
8 changes: 7 additions & 1 deletion pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -1834,11 +1834,17 @@
</exclusions>
</dependency>
<!-- Hive 2.3 need this to init Hive's FunctionRegistry -->
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-core-asl</artifactId>
<version>${codehaus.jackson.version}</version>
<scope>${hive.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<version>${codehaus.jackson.version}</version>
<scope>test</scope>
<scope>${hive.deps.scope}</scope>
</dependency>
<dependency>
<groupId>${hive.group}</groupId>
Expand Down
9 changes: 0 additions & 9 deletions sql/hive-thriftserver/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -147,15 +147,6 @@
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpcore</artifactId>
</dependency>
<!--
SPARK-44601: Add this test dependency to ensure that `hive-thriftserver` module
can be tested using Maven
-->
<dependency>
<groupId>org.codehaus.jackson</groupId>
<artifactId>jackson-mapper-asl</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<outputDirectory>target/scala-${scala.binary.version}/classes</outputDirectory>
Expand Down

0 comments on commit 7baf9da

Please sign in to comment.