[HotFix] Fix NoClassDefFoundError for Guava ImmutableMap during Hive-based Iceberg catalog initialization#4069
Conversation
amoro-ams/pom.xml
Outdated
| @@ -510,7 +510,6 @@ | |||
| <groupId>com.google.guava</groupId> | |||
| <artifactId>guava</artifactId> | |||
| <version>${guava.version}</version> | |||
There was a problem hiding this comment.
I'm encountering the same issue in the Paimon Hive Catalog. This creates a paradox with the current shared packages. Is there a better solution?
There was a problem hiding this comment.
Hi @czy006, thanks for the feedback!
This fix resolves the Paimon Hive Catalog case as well. The root cause is the same — Paimon's HiveCatalog uses unshaded HiveConf from the amoro-ams classpath, and HiveConf internally depends on unshaded Guava (ImmutableMap).
There's no shared package conflict:
- Amoro uses
org.apache.amoro.shade.guava32.* - Paimon uses
org.apache.paimon.shade.guava30.* - Hive/Iceberg uses
com.google.common.*(provided by this fix)
All three are separate namespaces — no collision.
If you're seeing a specific error that this fix doesn't cover, could you share the stack trace? Happy to dig into it together.
There was a problem hiding this comment.
This is a terrible dependency disaster, but there seems to be no better way to resolve package sharing conflicts. I suggest that this guava be version name to ${guava-hive.version} as it is related to the version of hive guava cc @j1wonpark
There was a problem hiding this comment.
Thanks @czy006! Updated the version property to ${guava-hive.version} as suggested.
|
@j1wonpark You need to rebase this PR, as your commits include content from previous PRs |
…ed Iceberg catalog initialization Signed-off-by: j1wonpark <jpark92@outlook.kr>
Signed-off-by: j1wonpark <jpark92@outlook.kr>
2f6a45e to
4a0eb23
Compare
|
@czy006 Done! Thanks |
Why are the changes needed?
When initializing a Hive Metastore-based Iceberg catalog in amoro-ams, a
NoClassDefFoundError: com/google/common/collect/ImmutableMapis thrown at runtime.Hive 3.1.3's
HiveConfdirectly references the original Guava classcom.google.common.collect.ImmutableMap, but the Guava dependency inamoro-ams/pom.xmlis declared with<scope>test</scope>, so it is excluded from the runtime classpath. All other Guava artifacts in the project are shaded (e.g.,amoro-shade-guava-32,iceberg-bundled-guava,flink-shaded-guava) and do not provide classes under the originalcom.google.common.*package. Additionally, the Hive dependency explicitly excludes Guava via<exclusion>, preventing transitive resolution.This issue also affects the official distribution and Docker images.
Brief change log
<scope>test</scope>from the Guava dependency inamoro-ams/pom.xmlso that it defaults tocompilescope and is included in the runtime classpath.How was this patch tested?
NoClassDefFoundError.Documentation