Supervisor RestartSupervisor saw failure [20]: A failure occurred while retrieving offsets. Shutting down. java.lang.Exception: A failure occurred while retrieving offsets. Shutting down. #503
Comments
|
I have a similar problem, does anyone know how to solve it? 2023-12-14 06:44:43,739 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-kafka-prod - Supervisor RestartSupervisor saw failure [7]: A failure occurred while retrieving offsets. Shutting down. java.lang.Exception: A failure occurred while retrieving offsets. Shutting down. |
|
I have the same issue running kafka-lag-exporter in a pod or locally onto the Kafka ec2 instance. `2023-12-14 15:58:37,895 DEBUG o.a.k.clients.admin.KafkaAdminClient akka://kafka-lag-exporter/user/consumer-group-collector-kafka.1-feature - [AdminClient clientId=adminclient-16] Max retries 0 for Call(callName=describeGroups(api=DESCRIBE_GROUPS), deadlineMs=1702569527790, tries=1, nextAllowedTryMs=1702569518895) reached 2023-12-14 15:58:37,897 DEBUG o.a.k.clients.admin.KafkaAdminClient - [AdminClient clientId=adminclient-16] Initiating close operation. Kafka version 3.2.2 and kafka-lag-exporter latest - 0.8.0 config kafka-lag-exporter { |
Kafka lag exporter return only cluster java metric
in prometheus
#HELP kafka_consumergroup_group_lag Group offset lag of a partition
#TYPE kafka_consumergroup_group_lag gauge
#HELP kafka_consumergroup_group_offset Last group consumed offset of a partition
#TYPE kafka_consumergroup_group_offset gauge
#HELP jvm_gc_collection_seconds Time spent in a given JVM garbage collector in seconds.
#TYPE jvm_gc_collection_seconds summary
jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 58716.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 182.253
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
#HELP kafka_consumergroup_group_max_lag Max group offset lag
#TYPE kafka_consumergroup_group_max_lag gauge
#HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously. jvm_gc_collection_seconds_count{gc="G1 Young Generation",} 58716.0
jvm_gc_collection_seconds_sum{gc="G1 Young Generation",} 182.253
jvm_gc_collection_seconds_count{gc="G1 Old Generation",} 0.0
jvm_gc_collection_seconds_sum{gc="G1 Old Generation",} 0.0
#HELP kafka_consumergroup_group_max_lag Max group offset lag
#TYPE kafka_consumergroup_group_max_lag gauge
#HELP jvm_memory_pool_allocated_bytes_total Total bytes allocated in a given JVM memory pool. Only updated after GC, not continuously.
#TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 2.6050688E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 1.0294682648E10
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 1.7257719660544E13
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1.730752E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 6.352273408E10
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 5864008.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 5.137648E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 4.2852736E7#TYPE jvm_memory_pool_allocated_bytes_total counter
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'profiled nmethods'",} 2.6050688E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Old Gen",} 1.0294682648E10
jvm_memory_pool_allocated_bytes_total{pool="G1 Eden Space",} 1.7257719660544E13
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-profiled nmethods'",} 1.730752E7
jvm_memory_pool_allocated_bytes_total{pool="G1 Survivor Space",} 6.352273408E10
jvm_memory_pool_allocated_bytes_total{pool="Compressed Class Space",} 5864008.0
jvm_memory_pool_allocated_bytes_total{pool="Metaspace",} 5.137648E7
jvm_memory_pool_allocated_bytes_total{pool="CodeHeap 'non-nmethods'",} 4.2852736E7
#HELP kafka_consumergroup_group_topic_sum_lag Sum of group offset lag across topic partitions
#TYPE kafka_consumergroup_group_topic_sum_lag gauge
#HELP jvm_threads_current Current thread count of a JVM
#TYPE jvm_threads_current gauge
jvm_threads_current 21.0
#HELP jvm_threads_daemon Daemon thread count of a JVM
#TYPE jvm_threads_daemon gauge
ERROR in log kafka_lag_exporter
2023-12-06 10:32:20,927 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-cluster - Supervisor RestartSupervisor saw failure [20]: A failure occurred while retrieving offsets. Shutting down. java.lang.Exception: A failure occurred while retrieving offsets. Shutting down.
at com.lightbend.kafkalagexporter.ConsumerGroupCollector$CollectorBehavior.$anonfun$collector$1(ConsumerGroupCollector.scala:302)
at akka.actor.typed.internal.BehaviorImpl$ReceiveBehavior.receive(BehaviorImpl.scala:136)
at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
at akka.actor.typed.internal.InterceptorImpl$$anon$2.apply(InterceptorImpl.scala:57)
at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:279)
at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:131)
at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:107)
at akka.actor.ActorCell.receiveMessage(ActorCell.scala:579)
at akka.actor.ActorCell.invoke(ActorCell.scala:547)
at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
at akka.dispatch.Mailbox.run(Mailbox.scala:231)
at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Exceeded maxRetries after 1 tries.
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2022)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:180)
at com.lightbend.kafkalagexporter.KafkaClient$.$anonfun$kafkaFuture$1(KafkaClient.scala:85)
at scala.concurrent.Future$.$anonfun$apply$1(Future.scala:659)
at scala.util.Success.$anonfun$map$1(Try.scala:255)
at scala.util.Success.map(Try.scala:213)
at scala.concurrent.Future.$anonfun$map$1(Future.scala:292)
at scala.concurrent.impl.Promise.liftedTree1$1(Promise.scala:33)
at scala.concurrent.impl.Promise.$anonfun$transform$1(Promise.scala:33)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:64)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: org.apache.kafka.common.errors.TimeoutException: Exceeded maxRetries after 1 tries.
Use kafka 3.2.3
Exporter 0.6.8
connection info in exporter log
2023-12-11 13:43:23,204 INFO o.a.k.c.consumer.ConsumerConfig akka://kafka-lag-exporter/user/consumer-group-collector-cluster - ConsumerConfig values:
allow.auto.create.topics = true
auto.commit.interval.ms = 5000
auto.offset.reset = latest
bootstrap.servers = [:9092]
check.crcs = true
client.dns.lookup = use_all_dns_ips
client.id = consumer-client-id
client.rack =
connections.max.idle.ms = 540000
default.api.timeout.ms = 60000
enable.auto.commit = false
exclude.internal.topics = true
fetch.max.bytes = 52428800
fetch.max.wait.ms = 500
fetch.min.bytes = 1
group.id = kafkalagexporter
group.instance.id = null
heartbeat.interval.ms = 3000
interceptor.classes = []
internal.leave.group.on.close = true
internal.throw.on.fetch.stable.offset.unsupported = false
isolation.level = read_uncommitted
key.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
max.partition.fetch.bytes = 1048576
max.poll.interval.ms = 300000
max.poll.records = 500
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partition.assignment.strategy = [class org.apache.kafka.clients.consumer.RangeAssignor, class org.apache.kafka.clients.consumer.CooperativeStickyAssignor]
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 60000
retry.backoff.ms = 1000
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.connect.timeout.ms = null
sasl.login.read.timeout.ms = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.login.retry.backoff.max.ms = 10000
sasl.login.retry.backoff.ms = 100
sasl.mechanism = GSSAPI
sasl.oauthbearer.clock.skew.seconds = 30
sasl.oauthbearer.expected.audience = null
sasl.oauthbearer.expected.issuer = null
sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
sasl.oauthbearer.jwks.endpoint.url = null
sasl.oauthbearer.scope.claim.name = scope
sasl.oauthbearer.sub.claim.name = sub
sasl.oauthbearer.token.endpoint.url = null
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
session.timeout.ms = 45000
socket.connection.setup.timeout.max.ms = 30000
socket.connection.setup.timeout.ms = 10000
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.certificate.chain = null
ssl.keystore.key = null
ssl.keystore.location = /app/prometheus/kafka/**.jks
ssl.keystore.password = [hidden]
ssl.keystore.type = JKS
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.certificates = null
ssl.truststore.location = /app/prometheus/kafka/*******.jks
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
value.deserializer = class org.apache.kafka.common.serialization.ByteArrayDeserializer
The text was updated successfully, but these errors were encountered: