-
Notifications
You must be signed in to change notification settings - Fork 29
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEATURE] Performance improvement #79
Comments
Hi @afillatre, |
It would be nice to run a performance analysis tool during the run to understand where the issue is. I cant' say for sure where the problem is ATM |
I faced the same issue with [INFO] -----------------< org.apache.accumulo:accumulo-test >------------------
[INFO] Building Apache Accumulo Testing 3.0.0-SNAPSHOT [15/17]
[INFO] from test/pom.xml
[INFO] --------------------------------[ jar ]---------------------------------
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-minicluster/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-minicluster/3.0.0-SNAPSHOT/maven-metadata.xml (792 B at 3.7 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-native/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-native/3.0.0-SNAPSHOT/maven-metadata.xml (790 B at 2.6 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-shell/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-shell/3.0.0-SNAPSHOT/maven-metadata.xml (786 B at 6.8 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-iterator-test-harness/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-iterator-test-harness/3.0.0-SNAPSHOT/maven-metadata.xml (802 B at 6.2 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compaction-coordinator/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compaction-coordinator/3.0.0-SNAPSHOT/maven-metadata.xml (803 B at 5.9 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compactor/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-compactor/3.0.0-SNAPSHOT/maven-metadata.xml (790 B at 8.8 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-gc/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-gc/3.0.0-SNAPSHOT/maven-metadata.xml (783 B at 3.0 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-manager/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-manager/3.0.0-SNAPSHOT/maven-metadata.xml (788 B at 7.2 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-monitor/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-monitor/3.0.0-SNAPSHOT/maven-metadata.xml (788 B at 6.5 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-server-base/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-server-base/3.0.0-SNAPSHOT/maven-metadata.xml (792 B at 2.1 kB/s)
Downloading from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-tserver/3.0.0-SNAPSHOT/maven-metadata.xml
Downloaded from apache.snapshots: https://repository.apache.org/snapshots/org/apache/accumulo/accumulo-tserver/3.0.0-SNAPSHOT/maven-metadata.xml (788 B at 10 kB/s)
[INFO]
[INFO] --- depclean:2.0.6:depclean (default-cli) @ accumulo-test ---
-------------------------------------------------------
[INFO] Starting DepClean dependency analysis
[INFO] Ignoring scope Scope(value=import)
[INFO] Ignoring scope Scope(value=runtime)
[INFO] Ignoring scope Scope(value=system)
[INFO] Ignoring scope Scope(value=provided)
[INFO] Ignoring scope Scope(value=test)
-------------------------------------------------------
D E P C L E A N A N A L Y S I S R E S U L T S
-------------------------------------------------------
USED DIRECT DEPENDENCIES [43]:
org.apache.hadoop:hadoop-client-minicluster:3.3.5:compile (35 MB)
org.apache.hadoop:hadoop-client-runtime:3.3.5:runtime (28 MB)
org.apache.hadoop:hadoop-client-api:3.3.5:compile (18 MB)
org.apache.accumulo:accumulo-core:3.0.0-20230426.091822-78:compile (6 MB)
org.bouncycastle:bcprov-jdk15on:1.70:compile (5 MB)
io.micrometer:micrometer-registry-statsd:1.10.6:compile (5 MB)
org.apache.curator:curator-test:5.3.0:compile (2 MB)
com.google.guava:guava:31.1-jre:compile (2 MB)
org.apache.zookeeper:zookeeper:3.8.1:compile (1 MB)
org.jline:jline:3.22.0:compile (1 MB)
org.bouncycastle:bcpkix-jdk15on:1.70:compile (941 KB)
io.micrometer:micrometer-core:1.10.6:compile (760 KB)
com.github.ben-manes.caffeine:caffeine:3.1.4:compile (734 KB)
org.apache.accumulo:accumulo-monitor:3.0.0-20230426.091823-78:compile (716 KB)
org.apache.accumulo:accumulo-server-base:3.0.0-20230426.091823-78:compile (644 KB)
org.apache.commons:commons-configuration2:2.8.0:compile (617 KB)
org.apache.commons:commons-lang3:3.12.0:compile (573 KB)
org.easymock:easymock:4.3:compile (563 KB)
org.apache.accumulo:accumulo-tserver:3.0.0-20230426.091824-78:compile (537 KB)
org.apache.thrift:libthrift:0.17.0:compile (345 KB)
org.apache.logging.log4j:log4j-1.2-api:2.20.0:compile (336 KB)
commons-io:commons-io:2.11.0:compile (319 KB)
org.apache.accumulo:accumulo-manager:3.0.0-20230426.091823-78:compile (287 KB)
org.apache.accumulo:accumulo-shell:3.0.0-20230426.091823-78:compile (272 KB)
com.google.code.gson:gson:2.9.1:compile (258 KB)
org.apache.zookeeper:zookeeper-jute:3.8.1:compile (250 KB)
org.junit.jupiter:junit-jupiter-engine:5.9.2:compile (240 KB)
org.junit.jupiter:junit-jupiter-api:5.9.2:compile (202 KB)
org.apache.hadoop:hadoop-distcp:3.3.5:compile (154 KB)
io.opentelemetry:opentelemetry-api:1.25.0:compile (126 KB)
com.beust:jcommander:1.82:compile (86 KB)
org.apache.accumulo:accumulo-minicluster:3.0.0-20230426.091823-78:compile (82 KB)
org.slf4j:slf4j-api:2.0.6:compile (61 KB)
commons-cli:commons-cli:1.5.0:compile (56 KB)
org.apache.accumulo:accumulo-gc:3.0.0-20230426.091822-78:compile (48 KB)
io.opentelemetry:opentelemetry-context:1.25.0:compile (46 KB)
org.apache.accumulo:accumulo-compaction-coordinator:3.0.0-20230426.091822-78:compile (43 KB)
org.apache.accumulo:accumulo-iterator-test-harness:3.0.0-20230426.091823-78:test (33 KB)
org.apache.accumulo:accumulo-compactor:3.0.0-20230426.091822-78:compile (32 KB)
org.apache.logging.log4j:log4j-slf4j2-impl:2.20.0:test (25 KB)
org.apache.hadoop:hadoop-minikdc:3.3.5:compile (16 KB)
org.apache.accumulo:accumulo-start:3.0.0-20230426.091824-78:compile (16 KB)
com.google.auto.service:auto-service:1.0.1:compile (12 KB)
USED TRANSITIVE DEPENDENCIES [98]:
org.apache.commons:commons-math3:3.6.1:compile (2 MB)
org.xerial.snappy:snappy-java:1.1.8.4:compile (1 MB)
org.apache.logging.log4j:log4j-core:2.20.0:compile (1 MB)
org.freemarker:freemarker:2.3.32:compile (1 MB)
com.fasterxml.jackson.core:jackson-databind:2.14.2:compile (1 MB)
org.hibernate.validator:hibernate-validator:7.0.5.Final:runtime (1 MB)
org.glassfish.jersey.core:jersey-common:3.0.9:compile (1 MB)
org.glassfish.jersey.core:jersey-server:3.0.9:compile (934 KB)
org.apache.datasketches:datasketches-java:3.3.0:compile (812 KB)
org.eclipse.jetty:jetty-server:11.0.13:compile (777 KB)
org.javassist:javassist:3.29.2-GA:runtime (775 KB)
org.apache.commons:commons-collections4:4.4:compile (734 KB)
io.netty:netty-common:4.1.89.Final:compile (642 KB)
org.eclipse.jetty.toolchain:jetty-jakarta-servlet-api:5.0.2:compile (590 KB)
org.junit.jupiter:junit-jupiter-params:5.9.2:test (566 KB)
org.eclipse.jetty:jetty-util:11.0.13:compile (554 KB)
io.netty:netty-handler:4.1.89.Final:compile (532 KB)
io.netty:netty-transport:4.1.89.Final:compile (476 KB)
org.bouncycastle:bcutil-jdk15on:1.70:compile (471 KB)
com.fasterxml.jackson.core:jackson-core:2.14.2:compile (448 KB)
junit:junit:4.13.2:compile (375 KB)
commons-codec:commons-codec:1.15:compile (345 KB)
io.netty:netty-codec:4.1.89.Final:compile (337 KB)
org.yaml:snakeyaml:1.33:compile (323 KB)
org.apache.logging.log4j:log4j-api:2.20.0:compile (305 KB)
jakarta.servlet:jakarta.servlet-api:5.0.0:compile (300 KB)
io.netty:netty-buffer:4.1.89.Final:compile (297 KB)
org.glassfish.jersey.core:jersey-client:3.0.9:compile (261 KB)
org.eclipse.jetty:jetty-http:11.0.13:compile (243 KB)
org.apache.commons:commons-text:1.10.0:compile (232 KB)
org.apache.kerby:kerb-core:1.0.1:compile (221 KB)
org.checkerframework:checker-qual:3.26.0:compile (217 KB)
org.eclipse.jetty:jetty-io:11.0.13:compile (211 KB)
org.apache.kerby:kerby-pkix:1.0.1:compile (199 KB)
org.glassfish.hk2:hk2-locator:3.0.3:runtime (199 KB)
org.glassfish.hk2:hk2-api:3.0.3:compile (196 KB)
org.junit.platform:junit-platform-engine:1.9.2:compile (184 KB)
org.hdrhistogram:HdrHistogram:2.1.12:runtime (169 KB)
org.glassfish:jakarta.el:4.0.2:runtime (169 KB)
org.eclipse.jetty:jetty-servlet:11.0.13:compile (162 KB)
io.netty:netty-transport-classes-epoll:4.1.89.Final:compile (141 KB)
jakarta.ws.rs:jakarta.ws.rs-api:3.0.0:compile (135 KB)
io.dropwizard.metrics:metrics-core:3.2.5:compile (133 KB)
org.glassfish.hk2:hk2-utils:3.0.3:compile (128 KB)
org.apache.datasketches:datasketches-memory:2.1.0:compile (126 KB)
jakarta.xml.bind:jakarta.xml.bind-api:3.0.1:compile (125 KB)
org.eclipse.jetty:jetty-security:11.0.13:compile (118 KB)
org.apache.kerby:kerb-crypto:1.0.1:compile (113 KB)
org.apache.kerby:kerb-client:1.0.1:compile (110 KB)
com.google.auto:auto-common:1.2:compile (108 KB)
org.junit.platform:junit-platform-commons:1.9.2:compile (100 KB)
org.apache.kerby:kerby-asn1:1.0.1:compile (99 KB)
jakarta.validation:jakarta.validation-api:3.0.0:compile (90 KB)
org.glassfish.jersey.ext:jersey-entity-filtering:3.0.9:compile (81 KB)
org.apache.kerby:kerb-server:1.0.1:compile (80 KB)
org.apache.kerby:kerb-admin:1.0.1:compile (79 KB)
jakarta.el:jakarta.el-api:4.0.0:runtime (78 KB)
org.glassfish.jersey.media:jersey-media-json-jackson:3.0.9:compile (78 KB)
org.glassfish.jersey.inject:jersey-hk2:3.0.9:runtime (77 KB)
com.fasterxml.jackson.core:jackson-annotations:2.14.2:compile (74 KB)
org.glassfish.jersey.containers:jersey-container-servlet-core:3.0.9:compile (71 KB)
org.glassfish.jaxb:txw2:4.0.2:runtime (71 KB)
com.fasterxml:classmate:1.5.1:runtime (66 KB)
org.apache.kerby:kerb-common:1.0.1:compile (63 KB)
io.micrometer:micrometer-observation:1.10.6:compile (63 KB)
org.jboss.logging:jboss-logging:3.5.0.Final:runtime (61 KB)
com.sun.activation:jakarta.activation:2.0.1:compile (60 KB)
commons-logging:commons-logging:1.2:compile (60 KB)
org.glassfish.jersey.ext:jersey-bean-validation:3.0.9:runtime (58 KB)
org.glassfish.jersey.ext:jersey-mvc:3.0.9:compile (52 KB)
org.objenesis:objenesis:3.3:compile (48 KB)
org.hamcrest:hamcrest-core:1.3:compile (43 KB)
io.netty:netty-transport-native-unix-common:4.1.89.Final:compile (42 KB)
org.apache.kerby:kerby-util:1.0.1:compile (39 KB)
io.micrometer:micrometer-commons:1.10.6:compile (38 KB)
io.netty:netty-transport-native-epoll:4.1.89.Final:runtime (38 KB)
io.netty:netty-resolver:4.1.89.Final:compile (36 KB)
io.opentelemetry:opentelemetry-semconv:1.25.0-alpha:compile (36 KB)
org.apache.kerby:kerb-util:1.0.1:compile (35 KB)
org.apache.logging.log4j:log4j-web:2.20.0:compile (34 KB)
com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.14.2:compile (31 KB)
org.apache.kerby:kerby-config:1.0.1:compile (29 KB)
org.latencyutils:LatencyUtils:2.0.3:runtime (29 KB)
org.glassfish.hk2.external:aopalliance-repackaged:3.0.3:compile (26 KB)
jakarta.annotation:jakarta.annotation-api:2.0.0:compile (24 KB)
org.glassfish.jersey.ext:jersey-mvc-freemarker:3.0.9:compile (23 KB)
org.apache.yetus:audience-annotations:0.12.0:compile (20 KB)
org.apache.kerby:kerb-simplekdc:1.0.1:compile (19 KB)
org.apache.kerby:kerb-identity:1.0.1:compile (19 KB)
com.google.code.findbugs:jsr305:3.0.2:compile (19 KB)
org.glassfish.hk2:osgi-resource-locator:1.0.3:compile (19 KB)
com.google.errorprone:error_prone_annotations:2.15.0:compile (15 KB)
jakarta.inject:jakarta.inject-api:2.0.0:compile (9 KB)
com.google.j2objc:j2objc-annotations:1.3:compile (8 KB)
org.opentest4j:opentest4j:1.2.0:compile (7 KB)
org.apiguardian:apiguardian-api:1.1.2:compile (6 KB)
com.google.guava:failureaccess:1.0.1:compile (4 KB)
com.google.auto.service:auto-service-annotations:1.0.1:compile (3 KB)
USED INHERITED DIRECT DEPENDENCIES [1]:
com.github.spotbugs:spotbugs-annotations:4.7.3:compile (14 KB)
USED INHERITED TRANSITIVE DEPENDENCIES [0]:
POTENTIALLY UNUSED DIRECT DEPENDENCIES [1]:
org.apache.accumulo:accumulo-native:3.0.0-20230426.091823-78:compile (13 KB)
POTENTIALLY UNUSED TRANSITIVE DEPENDENCIES [4]:
org.apache.kerby:kerby-xdr:1.0.1:compile (28 KB)
org.apache.kerby:token-provider:1.0.1:compile (18 KB)
io.netty:netty-transport-native-epoll:4.1.89.Final:compile (5 KB)
com.google.guava:listenablefuture:9999.0-empty-to-avoid-conflict-with-guava:compile (2 KB)
POTENTIALLY UNUSED INHERITED DIRECT DEPENDENCIES [0]:
POTENTIALLY UNUSED INHERITED TRANSITIVE DEPENDENCIES [0]:
[INFO] Analysis done in 525min 3s Summary for complete accumulo project Command to reproduce: mvn se.kth.castor:depclean-maven-plugin:2.0.6:depclean -DfailIfUnusedDirect=false -DfailIfUnusedTransitive=false -DignoreScopes=provided,test,runtime,system,import [INFO] --- depclean:2.0.6:depclean (default-cli) @ accumulo ---
-------------------------------------------------------
[INFO] Starting DepClean dependency analysis
[INFO] Skipping because packaging type is pom
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Accumulo Project 3.0.0-SNAPSHOT:
[INFO]
[INFO] Apache Accumulo Project ............................ SUCCESS [ 0.684 s]
[INFO] Apache Accumulo Start .............................. SUCCESS [ 13.324 s]
[INFO] Apache Accumulo Core ............................... SUCCESS [09:42 min]
[INFO] Apache Accumulo Server Base ........................ SUCCESS [20:34 min]
[INFO] Apache Accumulo Compaction Coordinator ............. SUCCESS [21:55 min]
[INFO] Apache Accumulo Compactor .......................... SUCCESS [21:32 min]
[INFO] Apache Accumulo GC Server .......................... SUCCESS [21:53 min]
[INFO] Apache Accumulo Manager Server ..................... SUCCESS [21:48 min]
[INFO] Apache Accumulo Monitor Server ..................... SUCCESS [22:17 min]
[INFO] Apache Accumulo Tablet Server ...................... SUCCESS [22:46 min]
[INFO] Apache Accumulo MiniCluster ........................ SUCCESS [23:00 min]
[INFO] Apache Accumulo Native Libraries ................... SUCCESS [ 0.023 s]
[INFO] Apache Accumulo Shell .............................. SUCCESS [ 02:58 h]
[INFO] Apache Accumulo Iterator Test Harness .............. SUCCESS [27:17 min]
[INFO] Apache Accumulo Testing ............................ SUCCESS [ 08:45 h]
[INFO] Apache Accumulo Hadoop MapReduce ................... SUCCESS [31:15 min]
[INFO] Apache Accumulo .................................... SUCCESS [ 0.489 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:47 h
[INFO] Finished at: 2023-04-27T09:56:22+02:00
[INFO] ------------------------------------------------------------------------ |
Is your feature request related to a problem? Please describe.
On large project, depclean takes a very long time to execute, most of the time is spent on excluding dependencies.
Here are some details. Given this analysis result:
It took 5h29min to complete depclean debloated-pom generation.
I have no timestamps, but I'd say it was between 20/30 min to analyse the dependencies, then the rest to exclude them and generate the debloated pom
Describe the solution you'd like
Better perfs, so this can be included in some regular CI jobs
Describe alternatives you've considered
None ATM
The text was updated successfully, but these errors were encountered: