Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug] Add the dependency of RoaringBitmap in paimon-micro-benchmarks. #4859

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[Bug] Add the dependency of RoaringBitmap in paimon-micro-benchmarks. #4859

wants to merge 1 commit into from

Conversation

Cribbee
Copy link

@Cribbee Cribbee commented Jan 8, 2025

Purpose

Fix the compilation error when packaging paimon-1.1 in moudle “paimon-micro-benchmarks”.

[INFO] --- maven-compiler-plugin:3.8.0:testCompile (default-testCompile) @ paimon-micro-benchmarks ---
[INFO] Compiling 11 source files to /paimon/paimon-benchmark/paimon-micro-benchmarks/target/test-classes
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /paimon/paimon-benchmark/paimon-micro-benchmarks/src/test/java/org/apache/paimon/benchmark/bitmap/RoaringBitmapBenchmark.java:[26,25] package org.roaringbitmap does not exist
[INFO] 1 error

Tests

API and Format

Documentation

@LinMingQiang
Copy link
Contributor

I didn't encounter this problem. Can you provide the steps to reproduce?

@Cribbee
Copy link
Author

Cribbee commented Jan 8, 2025

@LinMingQiang Hi，Here's how I reproduce it：
git clone -b master https://github.com/apache/paimon.git
cd paimon-master
export JAVA_HOME=./java8
export MAVEN_HOME=./apache-maven-3.8.5
$MAVEN_HOME/bin/mvn clean install -DskipTests -Dcheckstyle.skip=true -Dspotless.check.skip=true 1>$LOG_FILE 2>&1 &

[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /paimon-master/paimon-benchmark/paimon-micro-benchmarks/src/test/java/org/apache/paimon/benchmark/bitmap/RoaringBitmapBenchmark.java:[26,25] package org.roaringbitmap does not exist
[INFO] 1 error
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Paimon : 1.1-SNAPSHOT:
[INFO] 
[INFO] Paimon : ........................................... SUCCESS [  3.539 s]
[INFO] Paimon : Test utils ................................ SUCCESS [  2.040 s]
[INFO] Paimon : Common .................................... SUCCESS [ 29.846 s]
[INFO] Paimon : Benchmark ................................. SUCCESS [  0.068 s]
[INFO] Paimon : Benchmark : Cluster ....................... SUCCESS [  7.718 s]
[INFO] Paimon : Code Gen .................................. SUCCESS [ 24.189 s]
[INFO] Paimon : Code Gen Loader ........................... SUCCESS [  2.113 s]
[INFO] Paimon : Format .................................... SUCCESS [ 11.844 s]
[INFO] Paimon : Core ...................................... SUCCESS [ 11.559 s]
[INFO] Paimon : FileSystems : ............................. SUCCESS [  0.047 s]
[INFO] Paimon : FileSystems : S3 : Impl ................... SUCCESS [  6.138 s]
[INFO] Paimon : FileSystems : S3 .......................... SUCCESS [ 14.404 s]
[INFO] Paimon : Hive ...................................... SUCCESS [  0.242 s]
[INFO] Paimon : Hive Common ............................... SUCCESS [  1.653 s]
[INFO] Paimon : Hive Catalog .............................. SUCCESS [  2.792 s]
[INFO] Paimon : Bundle .................................... SUCCESS [  6.427 s]
[INFO] Paimon : Benchmark : Micro ......................... FAILURE [  0.621 s]
[INFO] Paimon : Service ................................... SKIPPED
[INFO] Paimon : Service : Client .......................... SKIPPED
[INFO] Paimon : Service : Runtime ......................... SKIPPED
[INFO] Paimon : Flink ..................................... SKIPPED
[INFO] Paimon : Flink : Common ............................ SKIPPED
[INFO] Paimon : Flink : CDC ............................... SKIPPED
[INFO] Paimon : Flink : 1.15 .............................. SKIPPED
[INFO] Paimon : Flink : 1.16 .............................. SKIPPED
[INFO] Paimon : Flink : 1.17 .............................. SKIPPED
[INFO] Paimon : Flink : 1.18 .............................. SKIPPED
[INFO] Paimon : Flink : 1.19 .............................. SKIPPED
[INFO] Paimon : Flink : 1.20 .............................. SKIPPED
[INFO] Paimon : Flink : Action ............................ SKIPPED
[INFO] Paimon : Spark ..................................... SKIPPED
[INFO] Paimon : Spark : Common : 2.12 ..................... SKIPPED
[INFO] Paimon : Docs ...................................... SKIPPED
[INFO] Paimon : Hive Connector Common ..................... SKIPPED
[INFO] Paimon : Spark3 : Common ........................... SKIPPED
[INFO] Paimon : Spark : UT ................................ SKIPPED
[INFO] Paimon : Spark : 3.3 ............................... SKIPPED
[INFO] Paimon : Tools : CI : Java ......................... SKIPPED
[INFO] Paimon : End to End Tests .......................... SKIPPED
[INFO] Paimon : FileSystems : Hadoop Shaded ............... SKIPPED
[INFO] Paimon : FileSystems : OSS : Impl .................. SKIPPED
[INFO] Paimon : FileSystems : OSS ......................... SKIPPED
[INFO] Paimon : Flink : 1.17 .............................. SKIPPED
[INFO] Paimon : Flink : 1.18 .............................. SKIPPED
[INFO] Paimon : Flink : 1.19 .............................. SKIPPED
[INFO] Paimon : Flink : 1.20 .............................. SKIPPED
[INFO] Paimon : Flink : Action ............................ SKIPPED
[INFO] Paimon : Spark ..................................... SKIPPED
[INFO] Paimon : Spark : Common : 2.12 ..................... SKIPPED
[INFO] Paimon : Docs ...................................... SKIPPED
[INFO] Paimon : Hive Connector Common ..................... SKIPPED
[INFO] Paimon : Spark3 : Common ........................... SKIPPED
[INFO] Paimon : Spark : UT ................................ SKIPPED
[INFO] Paimon : Spark : 3.3 ............................... SKIPPED
[INFO] Paimon : Tools : CI : Java ......................... SKIPPED
[INFO] Paimon : End to End Tests .......................... SKIPPED
[INFO] Paimon : FileSystems : Hadoop Shaded ............... SKIPPED
[INFO] Paimon : FileSystems : OSS : Impl .................. SKIPPED
[INFO] Paimon : FileSystems : OSS ......................... SKIPPED
[INFO] Paimon : Hive Connector : 2.1 ...................... SKIPPED
[INFO] Paimon : Hive Connector : 2.1-cdh-6.3 .............. SKIPPED
[INFO] Paimon : Hive Connector : 2.2 ...................... SKIPPED
[INFO] Paimon : Hive Connector : 2.3 ...................... SKIPPED
[INFO] Paimon : Hive Connector : 3.1 ...................... SKIPPED
[INFO] Paimon : Arrow ..................................... SKIPPED
[INFO] Paimon : Open API .................................. SKIPPED
[INFO] Paimon : Spark : 3.5 ............................... SKIPPED
[INFO] Paimon : Spark : 3.4 ............................... SKIPPED
[INFO] Paimon : Spark : 3.2 ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  02:05 min
[INFO] Finished at: 2025-01-08T13:14:47+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:testCompile (default-testCompile) on project paimon-micro-benchmarks: Compilation failure
[ERROR] /paimon-master/paimon-benchmark/paimon-micro-benchmarks/src/test/java/org/apache/paimon/benchmark/bitmap/RoaringBitmapBenchmark.java:[26,25] package org.roaringbitmap does not exist
[ERROR]
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :paimon-micro-benchmarks

@LinMingQiang
Copy link
Contributor

Can't reproduce in my environment, and check the dependency tree of the paimon-micro-benchmarks then dependency Maven: org.roaringbitmap:RoaringBitmap:1.0.5 can be found.

@Cribbee
Copy link
Author

Cribbee commented Jan 8, 2025

@LinMingQiang I tried to perform under the 'mvn dependency:tree' and not found org. Roaringbitmap: roaringbitmap: jar rely on, maybe something wrong in my env? But it will compile after adding the dependency to the pom file.

cd /paimon-master/paimon-benchmark/paimon-micro-benchmarks
mvn dependency:tree
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Paimon : Benchmark : Micro 1.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:3.1.1:tree (default-cli) @ paimon-micro-benchmarks ---
[INFO] org.apache.paimon:paimon-micro-benchmarks:jar:1.1-SNAPSHOT
[INFO] +- org.apache.logging.log4j:log4j-slf4j-impl:jar:2.17.1:compile
[INFO] +- org.apache.logging.log4j:log4j-api:jar:2.17.1:compile
[INFO] +- org.apache.logging.log4j:log4j-core:jar:2.17.1:compile
[INFO] +- org.apache.logging.log4j:log4j-1.2-api:jar:2.17.1:compile
[INFO] +- org.apache.hadoop:hadoop-hdfs-client:jar:2.8.5:compile
[INFO] |  \- com.squareup.okhttp:okhttp:jar:2.4.0:compile
[INFO] |     \- com.squareup.okio:okio:jar:1.4.0:compile
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.8.5:compile
[INFO] |  +- org.apache.hadoop:hadoop-annotations:jar:2.8.5:compile
[INFO] |  +- com.google.guava:guava:jar:11.0.2:compile
[INFO] |  +- commons-cli:commons-cli:jar:1.2:compile
[INFO] |  +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] |  +- xmlenc:xmlenc:jar:0.52:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.5.2:compile
[INFO] |  |  \- org.apache.httpcomponents:httpcore:jar:4.4.4:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.4:compile
[INFO] |  +- commons-io:commons-io:jar:2.4:compile
[INFO] |  +- commons-net:commons-net:jar:3.1:compile
[INFO] |  +- commons-collections:commons-collections:jar:3.2.2:compile
[INFO] |  +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] |  +- org.mortbay.jetty:jetty:jar:6.1.26:compile
[INFO] |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
[INFO] |  +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26:compile
[INFO] |  +- javax.servlet.jsp:jsp-api:jar:2.1:runtime
[INFO] |  +- com.sun.jersey:jersey-core:jar:1.9:compile
[INFO] |  +- com.sun.jersey:jersey-json:jar:1.9:compile
[INFO] |  |  +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] |  |  +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] |  |  +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.3:compile
[INFO] |  |  \- org.codehaus.jackson:jackson-xc:jar:1.8.3:compile
[INFO] |  +- com.sun.jersey:jersey-server:jar:1.9:compile
[INFO] |  |  \- asm:asm:jar:3.1:compile
[INFO] |  +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] |  +- net.java.dev.jets3t:jets3t:jar:0.9.0:compile
[INFO] |  |  \- com.jamesmurty.utils:java-xmlbuilder:jar:0.4:compile
[INFO] |  +- commons-lang:commons-lang:jar:2.6:compile
[INFO] |  +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] |  |  +- commons-digester:commons-digester:jar:1.8:compile
[INFO] |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
[INFO] |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
[INFO] |  +- com.google.code.gson:gson:jar:2.2.4:compile
[INFO] |  +- org.apache.hadoop:hadoop-auth:jar:2.8.5:compile
[INFO] |  |  +- com.nimbusds:nimbus-jose-jwt:jar:4.41.1:compile
[INFO] |  |  |  +- com.github.stephenc.jcip:jcip-annotations:jar:1.0-1:compile
[INFO] |  |  |  \- net.minidev:json-smart:jar:2.4.9:compile
[INFO] |  |  |     \- net.minidev:accessors-smart:jar:2.4.9:compile
[INFO] |  |  |        \- org.ow2.asm:asm:jar:9.3:compile
[INFO] |  |  +- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
[INFO] |  |  |  +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
[INFO] |  |  |  +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
[INFO] |  |  |  \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
[INFO] |  |  \- org.apache.curator:curator-framework:jar:2.7.1:compile
[INFO] |  +- com.jcraft:jsch:jar:0.1.54:compile
[INFO] |  +- org.apache.curator:curator-client:jar:2.7.1:compile
[INFO] |  +- org.apache.curator:curator-recipes:jar:2.7.1:compile
[INFO] |  +- org.apache.htrace:htrace-core4:jar:4.0.1-incubating:compile
[INFO] |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
[INFO] |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] |     \- org.tukaani:xz:jar:1.0:compile
[INFO] +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.8.5:compile
[INFO] |  +- org.apache.hadoop:hadoop-yarn-common:jar:2.8.5:compile
[INFO] |  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.8.5:compile
[INFO] |  |  +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
[INFO] |  |  |  +- javax.xml.stream:stax-api:jar:1.0-2:compile
[INFO] |  |  |  \- javax.activation:activation:jar:1.1:compile
[INFO] |  |  +- com.sun.jersey:jersey-client:jar:1.9:compile
[INFO] |  |  +- com.google.inject:guice:jar:3.0:compile
[INFO] |  |  |  +- javax.inject:javax.inject:jar:1:compile
[INFO] |  |  |  \- aopalliance:aopalliance:jar:1.0:compile
[INFO] |  |  \- com.sun.jersey.contribs:jersey-guice:jar:1.9:compile
[INFO] |  +- com.google.inject.extensions:guice-servlet:jar:3.0:compile
[INFO] |  \- io.netty:netty:jar:3.6.2.Final:compile
[INFO] +- org.xerial.snappy:snappy-java:jar:1.1.8.4:compile
[INFO] +- org.apache.avro:avro:jar:1.11.3:compile
[INFO] |  +- com.fasterxml.jackson.core:jackson-core:jar:2.14.2:compile
[INFO] |  \- com.fasterxml.jackson.core:jackson-databind:jar:2.14.2:compile
[INFO] |     \- com.fasterxml.jackson.core:jackson-annotations:jar:2.14.2:compile
[INFO] +- org.apache.commons:commons-lang3:jar:3.12.0:provided
[INFO] +- org.apache.paimon:paimon-bundle:jar:1.1-SNAPSHOT:compile
[INFO] |  \- org.lz4:lz4-java:jar:1.8.0:compile
[INFO] +- org.apache.paimon:paimon-test-utils:jar:1.1-SNAPSHOT:test
[INFO] |  +- org.junit.vintage:junit-vintage-engine:jar:5.8.1:test
[INFO] |  |  +- org.junit.platform:junit-platform-engine:jar:1.8.1:test
[INFO] |  |  +- junit:junit:jar:4.13.2:test
[INFO] |  |  |  \- org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] |  |  \- org.apiguardian:apiguardian-api:jar:1.1.2:test
[INFO] |  \- org.testcontainers:testcontainers:jar:1.19.1:test
[INFO] |     +- org.rnorth.duct-tape:duct-tape:jar:1.0.8:test
[INFO] |     |  \- org.jetbrains:annotations:jar:17.0.0:test
[INFO] |     +- com.github.docker-java:docker-java-api:jar:3.3.3:test
[INFO] |     \- com.github.docker-java:docker-java-transport-zerodep:jar:3.3.3:test
[INFO] |        +- com.github.docker-java:docker-java-transport:jar:3.3.3:test
[INFO] |        \- net.java.dev.jna:jna:jar:5.12.1:test
[INFO] +- org.slf4j:slf4j-api:jar:1.7.32:compile
[INFO] +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] +- org.junit.jupiter:junit-jupiter:jar:5.8.1:test
[INFO] |  +- org.junit.jupiter:junit-jupiter-api:jar:5.8.1:test
[INFO] |  |  +- org.opentest4j:opentest4j:jar:1.2.0:test
[INFO] |  |  \- org.junit.platform:junit-platform-commons:jar:1.8.1:test
[INFO] |  +- org.junit.jupiter:junit-jupiter-params:jar:5.8.1:test
[INFO] |  \- org.junit.jupiter:junit-jupiter-engine:jar:5.8.1:test
[INFO] \- org.assertj:assertj-core:jar:3.23.1:test
[INFO]    \- net.bytebuddy:byte-buddy:jar:1.12.10:test
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.798s
[INFO] Finished at: Wed Jan 08 14:15:42 CST 2025
[INFO] Final Memory: 21M/429M
[INFO] ------------------------------------------------------------------------

@LinMingQiang
Copy link
Contributor

521a36b0abd5c61c566d512c0b7e52cb

@Tan-JiaLiang
Copy link
Contributor

Can't reproduce in my environment +1. The roaringbitmap dependency is in the paimon-common module, and the paimon-common is wrap in the paimon-bundle. It is should not have such problem in theory. In addition, the roaringbitmap version has been upgrade to 1.2.1.

@JingsongLi
Copy link
Contributor

+1 Can't reproduce in my environment

@Cribbee
Copy link
Author

Cribbee commented Jan 8, 2025

@LinMingQiang @Tan-JiaLiang @JingsongLi Thax for your assistance~ I'll proceed to check my foundational environment.

@Cribbee Cribbee closed this Jan 8, 2025
@JingsongLi JingsongLi reopened this Jan 8, 2025
@JingsongLi
Copy link
Contributor

I really encountered it during partial compilation.. Can you extract a version number variable into the POM? For example, roaringbitmap.version.

@JingsongLi
Copy link
Contributor

@Cribbee

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Cribbee @LinMingQiang @Tan-JiaLiang @JingsongLi