Intermittent failure to load CSV data in batch mode due to an exception in index parallel sub task

[hkulkarniwhiz]

We are using batch mode to load CSV data off S3. While it works most of the time, we see intermittent failures in some datasources due to an exception seen in the middle manager's index parallel sub task log.

Affected Version

version 0.23

Description

The details on configuration are pasted after the exception section below -

The exception in the index parallel sub task log is as follows -

org.apache.druid.java.util.common.IAE: Received server error with status [400 Bad Request]; first 1KB of body: org.apache.druid.java.util.common.IAE: Could not find interval for timestamp [2018-05-11T00:00:00.000Z]
at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.findIntervalAndVersion(AbstractBatchIndexTask.java:764)
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseParallelIndexTaskRunner.findIntervalAndVersion(SinglePhaseParallelIndexTaskRunner.java:293)
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseParallelIndexTaskRunner.allocateNewSegment(SinglePhaseParallelIndexTaskRunner.java:243)
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.allocateSegment(ParallelIndexSupervisorTask.java:1240)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.s
at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:380) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.IndexTaskClient.submitSmileRequest(IndexTaskClient.java:272) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTaskClient.allocateSegment(ParallelIndexSupervisorTaskClient.java:92) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.SupervisorTaskCoordinatingSegmentAllocator.allocate(SupervisorTaskCoordinatingSegmentAllocator.java:60) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.getSegment(BaseAppenderatorDriver.java:338) [druid-server-0.23.0.jar:0.23.0]
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.append(BaseAppenderatorDriver.java:407) [druid-server-0.23.0.jar:0.23.0]
at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.add(BatchAppenderatorDriver.java:115) [druid-server-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:442) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.runTask(SinglePhaseSubTask.java:261) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:186) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-0.23.0.jar:0.23.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2024-07-05T06:04:38,841 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask - Encountered exception in parallel sub task.
org.apache.druid.java.util.common.IAE: Received server error with status [400 Bad Request]; first 1KB of body: org.apache.druid.java.util.common.IAE: Could not find interval for timestamp [2018-05-11T00:00:00.000Z]
at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.findIntervalAndVersion(AbstractBatchIndexTask.java:764)
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseParallelIndexTaskRunner.findIntervalAndVersion(SinglePhaseParallelIndexTaskRunner.java:293)
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseParallelIndexTaskRunner.allocateNewSegment(SinglePhaseParallelIndexTaskRunner.java:243)
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTask.allocateSegment(ParallelIndexSupervisorTask.java:1240)
at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
at com.s
at org.apache.druid.indexing.common.IndexTaskClient.submitRequest(IndexTaskClient.java:380) ~[druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.IndexTaskClient.submitSmileRequest(IndexTaskClient.java:272) ~[druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.batch.parallel.ParallelIndexSupervisorTaskClient.allocateSegment(ParallelIndexSupervisorTaskClient.java:92) ~[druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.SupervisorTaskCoordinatingSegmentAllocator.allocate(SupervisorTaskCoordinatingSegmentAllocator.java:60) ~[druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.getSegment(BaseAppenderatorDriver.java:338) ~[druid-server-0.23.0.jar:0.23.0]
at org.apache.druid.segment.realtime.appenderator.BaseAppenderatorDriver.append(BaseAppenderatorDriver.java:407) ~[druid-server-0.23.0.jar:0.23.0]
at org.apache.druid.segment.realtime.appenderator.BatchAppenderatorDriver.add(BatchAppenderatorDriver.java:115) ~[druid-server-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.generateAndPushSegments(SinglePhaseSubTask.java:442) ~[druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.batch.parallel.SinglePhaseSubTask.runTask(SinglePhaseSubTask.java:261) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.common.task.AbstractBatchIndexTask.run(AbstractBatchIndexTask.java:186) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:477) [druid-indexing-service-0.23.0.jar:0.23.0]
at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:449) [druid-indexing-service-0.23.0.jar:0.23.0]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_275]
2024-07-05T06:04:38,848 INFO [task-runner-0-priority-0] org.apache.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {
"id" : "single_phase_sub_task_care_gateway_onc_lnbjokfi_2024-07-05T06:04:23.010Z",
"status" : "FAILED",
"duration" : 12611,
"errorMsg" : "org.apache.druid.java.util.common.IAE: Received server error with status [400 Bad Request]; first 1K...",
"location" : {
"host" : null,
"port" : -1,
"tlsPort" : -1
}
}
2024-07-05T06:04:38,852 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [ANNOUNCEMENTS]
2024-07-05T06:04:38,854 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [SERVER]
2024-07-05T06:04:38,858 INFO [main] org.eclipse.jetty.server.AbstractConnector - Stopped ServerConnector@71fb1da3{HTTP/1.1, (http/1.1)}{0.0.0.0:8101}
2024-07-05T06:04:38,858 INFO [main] org.eclipse.jetty.server.session - node0 Stopped scavenging
2024-07-05T06:04:38,859 INFO [main] org.eclipse.jetty.server.handler.ContextHandler - Stopped o.e.j.s.ServletContextHandler@3831f4c2{/,null,STOPPED}
2024-07-05T06:04:38,868 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [NORMAL]
2024-07-05T06:04:38,868 INFO [main] org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner - Starting graceful shutdown of task[single_phase_sub_task_care_gateway_onc_lnbjokfi_2024-07-05T06:04:23.010Z].
2024-07-05T06:04:38,885 INFO [LookupExtractorFactoryContainerProvider-MainThread] org.apache.druid.query.lookup.LookupReferencesManager - Lookup Management loop exited. Lookup notices are not handled anymore.
2024-07-05T06:04:38,889 INFO [Curator-Framework-0] org.apache.curator.framework.imps.CuratorFrameworkImpl - backgroundOperationsLoop exiting
2024-07-05T06:04:38,994 INFO [main] org.apache.zookeeper.ZooKeeper - Session: 0x100000c7a3512f9 closed
2024-07-05T06:04:38,994 INFO [main-EventThread] org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100000c7a3512f9
2024-07-05T06:04:39,004 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Stopping lifecycle [module] stage [INIT]
Finished peon task

---

Configuration details -

024-08-02T06:10:23,109 INFO [main] org.apache.druid.cli.CliPeon - Starting up with processors[1], memory[12,884,901,888], maxMemory[12,884,901,888], directMemory[17,179,869,184]. Properties follow.
2024-08-02T06:10:23,109 INFO [main] org.apache.druid.cli.CliPeon - * aws.region: us-east-1
2024-08-02T06:10:23,109 INFO [main] org.apache.druid.cli.CliPeon - * awt.toolkit: sun.awt.X11.XToolkit
2024-08-02T06:10:23,109 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter: noop
2024-08-02T06:10:23,109 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.http.recipientBaseUrl: http://druid_exporter_url:druid_exporter_port/druid
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.emitter.logging.logLevel: error
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.expressions.useStrictBooleans: true
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.extensions.loadList: ["druid-s3-extensions", "druid-parquet-extensions", "druid-histogram", "druid-datasketches", "druid-lookups-cached-global", "postgresql-metadata-storage"]
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.generic.useDefaultValueForNull: false
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.global.http.compressionCodec: identity
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.global.http.eagerInitialization: false
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.host: 100.64.122.230
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.fork.property.druid.processing.numMergeBuffers: 2
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.directory: /opt/data/indexing-logs
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.logs.type: file
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.runner.javaOpts: -server -Xms1g -Xmx1g -XX:MaxDirectMemorySize=1g -Duser.timezone=UTC -Dfile.encoding=UTF-8 -XX:+ExitOnOutOfMemoryError -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.runner.javaOptsArray: ["-server", "-Xms12g", "-Xmx12g", "-XX:MaxDirectMemorySize=16g", "-Duser.timezone=UTC", "-Dfile.encoding=UTF-8", "-XX:+ExitOnOutOfMemoryError", "-Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager", "-XX:+UseG1GC", "-Daws.region=us-east-1"]
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.task.baseTaskDir: var/druid/task
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexer.task.hadoopWorkingPath: var/druid/hadoop-tmp
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.indexing.doubleStorage: double
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.lookup.enableLookupSyncOnStartup: false
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.connectURI: jdbc:postgresql://druid-preprod-postgresql:5432/druid
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.host: localhost
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.password:
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.port: 1527
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.connector.user: druid
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metadata.storage.type: postgresql
2024-08-02T06:10:23,110 INFO [main] org.apache.druid.cli.CliPeon - * druid.metrics.emitter.dimension.dataSource: care_gateway_onc
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.metrics.emitter.dimension.taskId: index_parallel_care_gateway_onc_aheeakel_2024-08-02T06:10:20.419Z
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.metrics.emitter.dimension.taskType: index_parallel
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.monitoring.monitors: ["org.apache.druid.java.util.metrics.JvmMonitor"]
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.plaintextPort: 8100
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.buffer.sizeBytes: 100000000
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.numMergeBuffers: 2
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.processing.numThreads: 1
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.selectors.coordinator.serviceName: druid/coordinator
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.selectors.indexing.serviceName: druid/overlord
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.server.hiddenProperties: ["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.server.http.numThreads: 60
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.service: druid/middleManager
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.sql.enable: true
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.sql.planner.useGroupingSetForExactDistinct: true
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.startup.logging.logProperties: true
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.storage.storageDirectory: var/druid/segments
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.storage.type: local
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.task.executor.enablePlaintextPort: true
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.task.executor.enableTlsPort: false
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.task.executor.host: 100.64.122.230
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.task.executor.plaintextPort: 8091
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.task.executor.service: druid/middleManager
2024-08-02T06:10:23,111 INFO [main] org.apache.druid.cli.CliPeon - * druid.task.executor.tlsPort: -1
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * druid.tlsPort: -1
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * druid.worker.capacity: 32
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * druid.worker.ip: localhost
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * druid.zk.paths.base: /druid
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * druid.zk.service.host: druid-preprod-zookeeper-headless:2181
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * file.encoding: UTF-8
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * file.encoding.pkg: sun.io
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * file.separator: /
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.awt.graphicsenv: sun.awt.X11GraphicsEnvironment
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.awt.printerjob: sun.print.PSPrinterJob
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.class.path: /tmp/conf/druid/cluster/_common:/tmp/conf/druid/cluster/data/middleManager:lib/FastInfoset-1.2.15.jar:lib/RoaringBitmap-0.9.0.jar:lib/accessors-smart-1.2.jar:lib/aether-api-0.9.0.M2.jar:lib/aether-connector-file-0.9.0.M2.jar:lib/aether-connector-okhttp-0.0.9.jar:lib/aether-impl-0.9.0.M2.jar:lib/aether-spi-0.9.0.M2.jar:lib/aether-util-0.9.0.M2.jar:lib/aggdesigner-algorithm-6.0.jar:lib/airline-2.8.4.jar:lib/airline-io-2.8.4.jar:lib/antlr4-runtime-4.5.1.jar:lib/aopalliance-1.0.jar:lib/asm-7.1.jar:lib/asm-analysis-7.1.jar:lib/asm-commons-7.1.jar:lib/asm-tree-7.1.jar:lib/async-http-client-2.5.3.jar:lib/async-http-client-netty-utils-2.5.3.jar:lib/audience-annotations-0.5.0.jar:lib/avatica-core-1.17.0.jar:lib/avatica-metrics-1.17.0.jar:lib/avatica-server-1.17.0.jar:lib/aws-java-sdk-core-1.12.37.jar:lib/aws-java-sdk-ec2-1.12.37.jar:lib/aws-java-sdk-kms-1.12.37.jar:lib/aws-java-sdk-s3-1.12.37.jar:lib/caffeine-2.8.0.jar:lib/calcite-core-1.21.0.jar:lib/calcite-linq4j-1.21.0.jar:lib/checker-qual-2.5.7.jar:lib/classmate-1.1.0.jar:lib/commons-beanutils-1.9.4.jar:lib/commons-codec-1.13.jar:lib/commons-collections-3.2.2.jar:lib/commons-collections4-4.2.jar:lib/commons-compiler-3.0.11.jar:lib/commons-compress-1.21.jar:lib/commons-dbcp2-2.0.1.jar:lib/commons-io-2.11.0.jar:lib/commons-lang-2.6.jar:lib/commons-lang3-3.8.1.jar:lib/commons-logging-1.1.1.jar:lib/commons-math3-3.6.1.jar:lib/commons-net-3.6.jar:lib/commons-pool2-2.2.jar:lib/commons-text-1.3.jar:lib/compress-lzf-1.0.4.jar:lib/config-magic-0.9.jar:lib/cron-scheduler-0.1.jar:lib/curator-client-4.3.0.jar:lib/curator-framework-4.3.0.jar:lib/curator-recipes-4.3.0.jar:lib/curator-x-discovery-4.3.0.jar:lib/datasketches-java-3.2.0.jar:lib/datasketches-memory-2.0.0.jar:lib/derby-10.14.2.0.jar:lib/derbyclient-10.14.2.0.jar:lib/derbynet-10.14.2.0.jar:lib/disruptor-3.3.6.jar:lib/druid-aws-common-0.23.0.jar:lib/druid-console-0.23.0.jar:lib/druid-core-0.23.0.jar:lib/druid-gcp-common-0.23.0.jar:lib/druid-hll-0.23.0.jar:lib/druid-indexing-hadoop-0.23.0.jar:lib/druid-indexing-service-0.23.0.jar:lib/druid-processing-0.23.0.jar:lib/druid-server-0.23.0.jar:lib/druid-services-0.23.0.jar:lib/druid-sql-0.23.0.jar:lib/error_prone_annotations-2.11.0.jar:lib/esri-geometry-api-2.2.0.jar:lib/extendedset-0.23.0.jar:lib/fastutil-8.5.4.jar:lib/fastutil-core-8.5.4.jar:lib/fastutil-extra-8.5.4.jar:lib/google-api-client-1.26.0.jar:lib/google-http-client-1.26.0.jar:lib/google-http-client-jackson2-1.26.0.jar:lib/google-oauth-client-1.26.0.jar:lib/guava-16.0.1.jar:lib/guice-4.1.0.jar:lib/guice-assistedinject-4.1.0.jar:lib/guice-multibindings-4.1.0.jar:lib/guice-servlet-4.1.0.jar:lib/hibernate-validator-5.2.5.Final.jar:lib/httpclient-4.5.13.jar:lib/httpcore-4.4.11.jar:lib/icu4j-55.1.jar:lib/ion-java-1.0.2.jar:lib/istack-commons-runtime-3.0.7.jar:lib/j2objc-annotations-1.1.jar:lib/jackson-annotations-2.10.5.jar:lib/jackson-core-2.10.5.jar:lib/jackson-databind-2.10.5.1.jar:lib/jackson-dataformat-cbor-2.10.5.jar:lib/jackson-dataformat-smile-2.10.5.jar:lib/janino-3.0.11.jar:lib/jackson-datatype-guava-2.10.5.jar:lib/jackson-datatype-joda-2.10.5.jar:lib/jackson-jaxrs-base-2.10.5.jar:lib/jackson-jaxrs-json-provider-2.10.5.jar:lib/jackson-jaxrs-smile-provider-2.10.5.jar:lib/jackson-jq-0.0.10.jar:lib/jackson-module-guice-2.10.5.jar:lib/jackson-module-jaxb-annotations-2.10.5.jar:lib/jakarta.activation-api-1.2.1.jar:lib/jakarta.inject-api-1.0.3.jar:lib/jakarta.xml.bind-api-2.3.2.jar:lib/javax.activation-1.2.0.jar:lib/javax.activation-api-1.2.0.jar:lib/javax.el-3.0.0.jar:lib/javax.el-api-3.0.0.jar:lib/javax.inject-1.jar:lib/javax.servlet-api-3.1.0.jar:lib/jaxb-api-2.3.1.jar:lib/jaxb-runtime-2.3.1.jar:lib/jboss-logging-3.2.1.Final.jar:lib/jcl-over-slf4j-1.7.36.jar:lib/jcodings-1.0.43.jar:lib/jdbi-2.63.1.jar:lib/jersey-core-1.19.4.jar:lib/jersey-guice-1.19.4.jar:lib/jersey-server-1.19.4.jar:lib/jersey-servlet-1.19.4.jar:lib/jetty-client-9.4.40.v20210413.jar:lib/jetty-continuation-9.4.40.v20210413.jar:lib/jetty-http-9.4.40.v20210413.jar:lib/jetty-io-9.4.40.v20210413.jar:lib/jetty-proxy-9.4.40.v20210413.jar:lib/jetty-rewrite-9.4.40.v20210413.jar:lib/jetty-security-9.4.40.v20210413.jar:lib/jetty-server-9.4.40.v20210413.jar:lib/jetty-servlet-9.4.40.v20210413.jar:lib/jetty-servlets-9.4.40.v20210413.jar:lib/jetty-util-9.4.40.v20210413.jar:lib/jetty-util-ajax-9.4.40.v20210413.jar:lib/jmespath-java-1.12.37.jar:lib/jna-4.5.1.jar:lib/joda-time-2.10.5.jar:lib/joni-2.1.27.jar:lib/json-path-2.3.0.jar:lib/json-smart-2.3.jar:lib/jsr305-2.0.1.jar:lib/jsr311-api-1.1.1.jar:lib/jvm-attach-api-1.5.jar:lib/log4j-1.2-api-2.17.1.jar:lib/log4j-api-2.17.1.jar:lib/log4j-core-2.17.1.jar:lib/log4j-jul-2.17.1.jar:lib/log4j-slf4j-impl-2.17.1.jar:lib/lz4-java-1.7.1.jar:lib/maven-aether-provider-3.1.1.jar:lib/maven-artifact-3.6.0.jar:lib/maven-model-3.1.1.jar:lib/maven-model-builder-3.1.1.jar:lib/maven-repository-metadata-3.1.1.jar:lib/maven-settings-3.1.1.jar:lib/maven-settings-builder-3.1.1.jar:lib/metrics-core-4.0.0.jar:lib/netty-3.10.6.Final.jar:lib/netty-buffer-4.1.68.Final.jar:lib/netty-codec-4.1.68.Final.jar:lib/netty-codec-dns-4.1.68.Final.jar:lib/netty-codec-http-4.1.68.Final.jar:lib/netty-codec-socks-4.1.68.Final.jar:lib/netty-common-4.1.68.Final.jar:lib/netty-handler-4.1.68.Final.jar:lib/netty-handler-proxy-4.1.68.Final.jar:lib/netty-reactive-streams-2.0.0.jar:lib/netty-resolver-4.1.68.Final.jar:lib/netty-resolver-dns-4.1.68.Final.jar:lib/netty-transport-4.1.68.Final.jar:lib/netty-transport-native-epoll-4.1.68.Final-linux-x86_64.jar:lib/netty-transport-native-unix-common-4.1.68.Final.jar:lib/okhttp-1.0.2.jar:lib/opencsv-4.6.jar:lib/plexus-interpolation-1.19.jar:lib/plexus-utils-3.0.24.jar:lib/protobuf-java-3.11.0.jar:lib/reactive-streams-1.0.2.jar:lib/resilience4j-bulkhead-1.3.1.jar:lib/resilience4j-core-1.3.1.jar:lib/rhino-1.7.11.jar:lib/shims-0.9.0.jar:lib/sigar-1.6.5.132.jar:lib/slf4j-api-1.7.36.jar:lib/spymemcached-2.12.3.jar:lib/stax-ex-1.8.jar:lib/tesla-aether-0.0.5.jar:lib/txw2-2.3.1.jar:lib/validation-api-1.1.0.Final.jar:lib/vavr-0.10.2.jar:lib/vavr-match-0.10.2.jar:lib/wagon-provider-api-2.4.jar:lib/xz-1.8.jar:lib/zookeeper-3.5.9.jar:lib/zookeeper-jute-3.5.9.jar:lib/zstd-jni-1.3.3-1.jar:
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.class.version: 52.0
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.endorsed.dirs: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/endorsed
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.ext.dirs: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/ext:/usr/java/packages/lib/ext
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.home: /usr/lib/jvm/java-8-openjdk-amd64/jre
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.io.tmpdir: var/tmp
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.library.path: /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.runtime.name: OpenJDK Runtime Environment
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.runtime.version: 1.8.0_275-8u275-b01-1~deb9u1-b01
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.name: Java Platform API Specification
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.vendor: Oracle Corporation
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.specification.version: 1.8
2024-08-02T06:10:23,112 INFO [main] org.apache.druid.cli.CliPeon - * java.util.logging.manager: org.apache.logging.log4j.jul.LogManager
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vendor: Oracle Corporation
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vendor.url: http://java.oracle.com/
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vendor.url.bug: http://bugreport.sun.com/bugreport/
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.version: 1.8.0_275
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.info: mixed mode
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.name: OpenJDK 64-Bit Server VM
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.name: Java Virtual Machine Specification
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.vendor: Oracle Corporation
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.specification.version: 1.8
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.vendor: Oracle Corporation
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * java.vm.version: 25.275-b01
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * line.separator:
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownCallbackRegistry: org.apache.druid.common.config.Log4jShutdown
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * log4j.shutdownHookEnabled: true
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * log4j2.configurationFactory: org.apache.druid.indexing.common.tasklogs.ConsoleLoggingEnforcementConfigurationFactory
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * log4j2.is.webapp: false
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * os.arch: amd64
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * os.name: Linux
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * os.version: 5.10.219-208.866.amzn2.x86_64
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * path.separator: :
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * sun.arch.data.model: 64
2024-08-02T06:10:23,113 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.class.path: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-8-openjdk-amd64/jre/classes
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.boot.library.path: /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.endian: little
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.cpu.isalist:
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.io.unicode.encoding: UnicodeLittle
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.command: org.apache.druid.cli.Main internal peon var/druid/task/index_parallel_care_gateway_onc_aheeakel_2024-08-02T06:10:20.419Z/task.json var/druid/task/index_parallel_care_gateway_onc_aheeakel_2024-08-02T06:10:20.419Z/b96ca5e9-4fc4-401f-9f1a-0dafd2093764/status.json var/druid/task/index_parallel_care_gateway_onc_aheeakel_2024-08-02T06:10:20.419Z/b96ca5e9-4fc4-401f-9f1a-0dafd2093764/report.json
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.java.launcher: SUN_STANDARD
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.jnu.encoding: UTF-8
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.management.compiler: HotSpot 64-Bit Tiered Compilers
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * sun.os.patch.level: unknown
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * user.dir: /opt/druid
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * user.home: /root
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * user.language: en
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * user.name: root
2024-08-02T06:10:23,114 INFO [main] org.apache.druid.cli.CliPeon - * user.timezone: UTC
2024-08-02T06:10:23,122 WARN [main] org.apache.druid.indexing.common.config.TaskConfig - Batch processing mode argument value is null or not valid:[null], defaulting to[CLOSED_SEGMENTS]
2024-08-02T06:10:23,155 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheMaxCount, ${base_path}.buffer.poolCacheMaxCount] on [org.apache.druid.query.DruidProcessingConfig#poolCacheMaxCount()]
2024-08-02T06:10:23,156 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.pool.awaitShutdownMillis] on [org.apache.druid.query.DruidProcessingConfig#getMergePoolAwaitShutdownMillis()]
2024-08-02T06:10:23,156 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.fifo] on [org.apache.druid.query.DruidProcessingConfig#isFifo()]
2024-08-02T06:10:23,156 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.tmpDir] on [org.apache.druid.query.DruidProcessingConfig#getTmpDir()]
2024-08-02T06:10:23,156 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [100000000] for [druid.processing.buffer.sizeBytes] on [org.apache.druid.query.DruidProcessingConfig#intermediateComputeSizeBytesConfigured()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [druid.computation.buffer.poolCacheInitialCount, ${base_path}.buffer.poolCacheInitialCount] on [org.apache.druid.query.DruidProcessingConfig#getNumInitalBuffersForIntermediatePool()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [1] for [druid.processing.numThreads] on [org.apache.druid.query.DruidProcessingConfig#getNumThreadsConfigured()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning value [2] for [druid.processing.numMergeBuffers] on [org.apache.druid.query.DruidProcessingConfig#getNumMergeBuffersConfigured()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.columnCache.sizeBytes] on [org.apache.druid.query.DruidProcessingConfig#columnCacheSizeBytes()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.useParallelMergePool] on [org.apache.druid.query.DruidProcessingConfig#useParallelMergePoolConfigured()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.pool.parallelism] on [org.apache.druid.query.DruidProcessingConfig#getMergePoolParallelismConfigured()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.pool.defaultMaxQueryParallelism] on [org.apache.druid.query.DruidProcessingConfig#getMergePoolDefaultMaxQueryParallelism()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.task.targetRunTimeMillis] on [org.apache.druid.query.DruidProcessingConfig#getMergePoolTargetTaskRunTimeMillis()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.task.initialYieldNumRows] on [org.apache.druid.query.DruidProcessingConfig#getMergePoolTaskInitialYieldRows()]
2024-08-02T06:10:23,157 INFO [main] org.skife.config.ConfigurationObjectFactory - Using method itself for [${base_path}.merge.task.smallBatchNumRows] on [org.apache.druid.query.DruidProcessingConfig#getMergePoolSmallBatchRows()]
2024-08-02T06:10:23,158 INFO [main] org.skife.config.ConfigurationObjectFactory - Assigning default value [processing-%s] for [${base_path}.formatString] on [org.apache.druid.java.util.common.concurrent.ExecutorServiceConfig#getFormatString()]
2024-08-02T06:10:23,200 INFO [main] org.apache.druid.segment.loading.SegmentLocalCacheManager - Using storage location strategy: [LeastBytesUsedStorageLocationSelectorStrategy]
2024-08-02T06:10:23,236 INFO [main] org.apache.druid.guice.StorageNodeModule - Segment cache not configured on ServerType [indexer-executor]. It will not be assignable for segment placement
2024-08-02T06:10:23,319 INFO [main] org.eclipse.jetty.util.log - Logging initialized @2869ms to org.eclipse.jetty.util.log.Slf4jLog
2024-08-02T06:10:23,329 INFO [main] org.apache.druid.server.initialization.jetty.JettyServerModule - Creating http connector with port [8100]
2024-08-02T06:10:23,410 WARN [main] org.eclipse.jetty.server.handler.gzip.GzipHandler - minGzipSize of 0 is inefficient for short content, break even is size 23
2024-08-02T06:10:23,429 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new intermediate processing buffer[0] of size[100,000,000]
2024-08-02T06:10:23,499 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[0] of size[100,000,000]
2024-08-02T06:10:23,532 INFO [main] org.apache.druid.offheap.OffheapBufferGenerator - Allocating new result merging buffer[1] of size[100,000,000]
2024-08-02T06:10:23,574 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [INIT]
2024-08-02T06:10:23,574 INFO [main] org.apache.druid.java.util.common.lifecycle.Lifecycle - Starting lifecycle [module] stage [NORMAL]
2024-08-02T06:10:23,575 INFO [main] org.apache.curator.framework.imps.CuratorFrameworkImpl - Starting
2024-08-02T06:10:23,579 INFO [main] org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=druid-preprod-zookeeper-headless:2181 sessionTimeout=30000 watcher=org.apache.curator.ConnectionState@5e7cd0df
2024-08-02T06:10:23,583 INFO [main] org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2024-08-02T06:10:23,586 INFO [main] org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 4194304 Bytes