KAFKA-topics.sh --list TimeoutException

[Ahmedkaramakka]

ERROR org.apache.kafka.common.errors.TimeoutException: Call(callName=listTopics, deadlineMs=1706136118301, tries=1, nextAllowedTryMs=1706136118444) timed out at 1706136118344 after 1 attempt(s)
Caused by: org.apache.kafka.common.errors.DisconnectException: Cancelled listTopics request with correlation id 3 due to node 1 being disconnected
(kafka.admin.TopicCommand$)

This the error I get when I try to list the commands
the history of the situation I have a server standalone VM on OCI ( oracle cloud ) this is the source and the destination is normal 3 linux servers as a cluster with apache kafka 3.6 , when I try to list i get timeout, but what is confusing me when I try to create or produce or consume it works well , there is a firewall between OCI and servers but network team confirms that everything is allowed.

• listing topics from the servers to OCI wasn't working also but when we changed the etc/hosts file in the servers, it could list the topics on OCI

• I will post below the debugging I tried when I ran the command

• Registered kafka:type=kafka.Log4jController MBean
  AdminClientConfig values:
  auto.include.jmx.reporter = true
  bootstrap.servers = [xx.x.xxx:9092, xx.x.xxx:9092]
  client.dns.lookup = use_all_dns_ips
  client.id =
  connections.max.idle.ms = 300000
  default.api.timeout.ms = 60000
  metadata.max.age.ms = 300000
  metric.reporters = []
  metrics.num.samples = 2
  metrics.recording.level = INFO
  metrics.sample.window.ms = 30000
  receive.buffer.bytes = 65536
  reconnect.backoff.max.ms = 1000
  reconnect.backoff.ms = 50
  request.timeout.ms = 30000
  retries = 2147483647
  retry.backoff.ms = 100
  sasl.client.callback.handler.class = null
  sasl.jaas.config = null
  sasl.kerberos.kinit.cmd = /usr/bin/kinit
  sasl.kerberos.min.time.before.relogin = 60000
  sasl.kerberos.service.name = null
  sasl.kerberos.ticket.renew.jitter = 0.05
  sasl.kerberos.ticket.renew.window.factor = 0.8
  sasl.login.callback.handler.class = null
  sasl.login.class = null
  sasl.login.connect.timeout.ms = null
  sasl.login.read.timeout.ms = null
  sasl.login.refresh.buffer.seconds = 300
  sasl.login.refresh.min.period.seconds = 60
  sasl.login.refresh.window.factor = 0.8
  sasl.login.refresh.window.jitter = 0.05
  sasl.login.retry.backoff.max.ms = 10000
  sasl.login.retry.backoff.ms = 100
  sasl.mechanism = GSSAPI
  sasl.oauthbearer.clock.skew.seconds = 30
  sasl.oauthbearer.expected.audience = null
  sasl.oauthbearer.expected.issuer = null
  sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
  sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
  sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
  sasl.oauthbearer.jwks.endpoint.url = null
  sasl.oauthbearer.scope.claim.name = scope
  sasl.oauthbearer.sub.claim.name = sub
  sasl.oauthbearer.token.endpoint.url = null
  security.protocol = PLAINTEXT
  security.providers = null
  send.buffer.bytes = 131072
  socket.connection.setup.timeout.max.ms = 30000
  socket.connection.setup.timeout.ms = 10000
  ssl.cipher.suites = null
  ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
  ssl.endpoint.identification.algorithm = https
  ssl.engine.factory.class = null
  ssl.key.password = null
  ssl.keymanager.algorithm = SunX509
  ssl.keystore.certificate.chain = null
  ssl.keystore.key = null
  ssl.keystore.location = null
  ssl.keystore.password = null
  ssl.keystore.type = JKS
  ssl.protocol = TLSv1.3
  ssl.provider = null
  ssl.secure.random.implementation = null
  ssl.trustmanager.algorithm = PKIX
  ssl.truststore.certificates = null
  ssl.truststore.location = null
  ssl.truststore.password = null
  ssl.truststore.type = JKS

[AdminClient clientId=adminclient-1] Setting bootstrap cluster metadata Cluster(id = null, nodes = [xx.x.xxx:9092 (id: -2 rack: null), xx.x.xxx:9092 (id: -1 rack: null)], partitions = [], controller = null).
Registered metric named MetricName [name=count, group=kafka-metrics-count, description=total number of registered metrics, tags={client-id=adminclient-1}]
Added sensor with name connections-closed:
Registered metric named MetricName [name=connection-close-total, group=admin-client-metrics, description=The total number of connections closed, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=connection-close-rate, group=admin-client-metrics, description=The number of connections closed per second, tags={client-id=adminclient-1}]
Added sensor with name connections-created:
Registered metric named MetricName [name=connection-creation-total, group=admin-client-metrics, description=The total number of new connections established, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=connection-creation-rate, group=admin-client-metrics, description=The number of new connections established per second, tags={client-id=adminclient-1}]
Added sensor with name successful-authentication:
Registered metric named MetricName [name=successful-authentication-total, group=admin-client-metrics, description=The total number of connections with successful authentication, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=successful-authentication-rate, group=admin-client-metrics, description=The number of connections with successful authentication per second, tags={client-id=adminclient-1}]
Added sensor with name successful-reauthentication:
Registered metric named MetricName [name=successful-reauthentication-total, group=admin-client-metrics, description=The total number of successful re-authentication of connections, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=successful-reauthentication-rate, group=admin-client-metrics, description=The number of successful re-authentication of connections per second, tags={client-id=adminclient-1}]
Added sensor with name successful-authentication-no-reauth:
Registered metric named MetricName [name=successful-authentication-no-reauth-total, group=admin-client-metrics, description=The total number of connections with successful authentication where the client does not support re-authentication, tags={client-id=adminclient-1}]
Added sensor with name failed-authentication:
Registered metric named MetricName [name=failed-authentication-total, group=admin-client-metrics, description=The total number of connections with failed authentication, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=failed-authentication-rate, group=admin-client-metrics, description=The number of connections with failed authentication per second, tags={client-id=adminclient-1}]
Added sensor with name failed-reauthentication:
Registered metric named MetricName [name=failed-reauthentication-total, group=admin-client-metrics, description=The total number of failed re-authentication of connections, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=failed-reauthentication-rate, group=admin-client-metrics, description=The number of failed re-authentication of connections per second, tags={client-id=adminclient-1}]
Added sensor with name reauthentication-latency:
Registered metric named MetricName [name=reauthentication-latency-max, group=admin-client-metrics, description=The max latency observed due to re-authentication, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=reauthentication-latency-avg, group=admin-client-metrics, description=The average latency observed due to re-authentication, tags={client-id=adminclient-1}]
Added sensor with name bytes-sent-received:
Registered metric named MetricName [name=network-io-total, group=admin-client-metrics, description=The total number of network operations (reads or writes) on all connections, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=network-io-rate, group=admin-client-metrics, description=The number of network operations (reads or writes) on all connections per second, tags={client-id=adminclient-1}]
Added sensor with name bytes-sent:
Registered metric named MetricName [name=outgoing-byte-total, group=admin-client-metrics, description=The total number of outgoing bytes sent to all servers, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=outgoing-byte-rate, group=admin-client-metrics, description=The number of outgoing bytes sent to all servers per second, tags={client-id=adminclient-1}]
Added sensor with name requests-sent:
Registered metric named MetricName [name=request-total, group=admin-client-metrics, description=The total number of requests sent, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=request-rate, group=admin-client-metrics, description=The number of requests sent per second, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=request-size-avg, group=admin-client-metrics, description=The average size of requests sent., tags={client-id=adminclient-1}]
Registered metric named MetricName [name=request-size-max, group=admin-client-metrics, description=The maximum size of any request sent., tags={client-id=adminclient-1}]
Added sensor with name bytes-received:
Registered metric named MetricName [name=incoming-byte-total, group=admin-client-metrics, description=The total number of bytes read off all sockets, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=incoming-byte-rate, group=admin-client-metrics, description=The number of bytes read off all sockets per second, tags={client-id=adminclient-1}]
Added sensor with name responses-received:
Registered metric named MetricName [name=response-total, group=admin-client-metrics, description=The total number of responses received, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=response-rate, group=admin-client-metrics, description=The number of responses received per second, tags={client-id=adminclient-1}]
Added sensor with name select-time:
Registered metric named MetricName [name=select-total, group=admin-client-metrics, description=The total number of times the I/O layer checked for new I/O to perform, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=select-rate, group=admin-client-metrics, description=The number of times the I/O layer checked for new I/O to perform per second, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=io-wait-time-ns-avg, group=admin-client-metrics, description=The average length of time the I/O thread spent waiting for a socket ready for reads or writes in nanoseconds., tags={client-id=adminclient-1}]
Registered metric named MetricName [name=io-waittime-total, group=admin-client-metrics, description=Deprecated The total time the I/O thread spent waiting, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=io-wait-ratio, group=admin-client-metrics, description=Deprecated The fraction of time the I/O thread spent waiting, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=io-wait-time-ns-total, group=admin-client-metrics, description=The total time the I/O thread spent waiting, tags={client-id=adminclient-1}]
Added sensor with name io-time:
Registered metric named MetricName [name=io-time-ns-avg, group=admin-client-metrics, description=The average length of time for I/O per select call in nanoseconds., tags={client-id=adminclient-1}]
Registered metric named MetricName [name=iotime-total, group=admin-client-metrics, description=Deprecated The total time the I/O thread spent doing I/O, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=io-ratio, group=admin-client-metrics, description=Deprecated The fraction of time the I/O thread spent doing I/O, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=io-time-ns-total, group=admin-client-metrics, description=The total time the I/O thread spent doing I/O, tags={client-id=adminclient-1}]
[AdminClient clientId=adminclient-1] sslCiphers: created new gauge suite with maxEntries = 100.
[AdminClient clientId=adminclient-1] clients: created new gauge suite with maxEntries = 100.
Registered metric named MetricName [name=connection-count, group=admin-client-metrics, description=The current number of active connections., tags={client-id=adminclient-1}]
Kafka version: 3.6.1
Kafka commitId: 5e3c2b738d253ff5
Kafka startTimeMs: 1706133998344
Registered metric named MetricName [name=version, group=app-info, description=Metric indicating version, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=commit-id, group=app-info, description=Metric indicating commit-id, tags={client-id=adminclient-1}]
Registered metric named MetricName [name=start-time-ms, group=app-info, description=Metric indicating start-time-ms, tags={client-id=adminclient-1}]
[AdminClient clientId=adminclient-1] Kafka admin client initialized
[AdminClient clientId=adminclient-1] Thread starting
[AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1706133998348
[AdminClient clientId=adminclient-1] Found least loaded node xx.x.xxx:9092 (id: -1 rack: null) with no active connection
[AdminClient clientId=adminclient-1] Assigned Call(callName=fetchMetadata, deadlineMs=1706134028348, tries=0, nextAllowedTryMs=0) to node xx.x.xxx:9092 (id: -1 rack: null)
Resolved host xx.x.xxx as xx.x.xxx
[AdminClient clientId=adminclient-1] Initiating connection to node xx.x.xxx:9092 (id: -1 rack: null) using address /xx.x.xxx
[AdminClient clientId=adminclient-1] Queueing Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) with a timeout 30000 ms from now.
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: -1 rack: null). Must delay 11950 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=11950)
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0)] at 1706133998358
[AdminClient clientId=adminclient-1] Metadata is not ready: we have not fetched metadata from the bootstrap nodes yet.
[AdminClient clientId=adminclient-1] Unable to assign Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) to a node.
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: -1 rack: null). Must delay 11950 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=100)
Added sensor with name node--1.requests-sent
Registered metric named MetricName [name=request-total, group=admin-client-node-metrics, description=The total number of requests sent, tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=request-rate, group=admin-client-node-metrics, description=The number of requests sent per second, tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=request-size-avg, group=admin-client-node-metrics, description=The average size of requests sent., tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=request-size-max, group=admin-client-node-metrics, description=The maximum size of any request sent., tags={client-id=adminclient-1, node-id=node--1}]
Added sensor with name node--1.bytes-sent
Registered metric named MetricName [name=outgoing-byte-total, group=admin-client-node-metrics, description=The total number of outgoing bytes, tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=outgoing-byte-rate, group=admin-client-node-metrics, description=The number of outgoing bytes per second, tags={client-id=adminclient-1, node-id=node--1}]
Added sensor with name node--1.responses-received
Registered metric named MetricName [name=response-total, group=admin-client-node-metrics, description=The total number of responses received, tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=response-rate, group=admin-client-node-metrics, description=The number of responses received per second, tags={client-id=adminclient-1, node-id=node--1}]
Added sensor with name node--1.bytes-received
Registered metric named MetricName [name=incoming-byte-total, group=admin-client-node-metrics, description=The total number of incoming bytes, tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=incoming-byte-rate, group=admin-client-node-metrics, description=The number of incoming bytes per second, tags={client-id=adminclient-1, node-id=node--1}]
Added sensor with name node--1.latency
Registered metric named MetricName [name=request-latency-avg, group=admin-client-node-metrics, description=, tags={client-id=adminclient-1, node-id=node--1}]
Registered metric named MetricName [name=request-latency-max, group=admin-client-node-metrics, description=, tags={client-id=adminclient-1, node-id=node--1}]
[AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -1
[AdminClient clientId=adminclient-1] Completed connection to node -1. Fetching API versions.
[AdminClient clientId=adminclient-1] Initiating API versions fetch from node -1.
[AdminClient clientId=adminclient-1] No version information found when sending API_VERSIONS with correlation id 0 to node -1. Assuming version 3.
[AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0, headerVersion=2) and timeout 3600000 to node -1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.6.1')
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0)] at 1706133998652
[AdminClient clientId=adminclient-1] Metadata is not ready: we have not fetched metadata from the bootstrap nodes yet.
[AdminClient clientId=adminclient-1] Unable to assign Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) to a node.
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: -1 rack: null). Must delay 9223372036854775807 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=100)
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0)] at 1706133998654
[AdminClient clientId=adminclient-1] Metadata is not ready: we have not fetched metadata from the bootstrap nodes yet.
[AdminClient clientId=adminclient-1] Unable to assign Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) to a node.
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: -1 rack: null). Must delay 9223372036854775807 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=100)
[AdminClient clientId=adminclient-1] Received API_VERSIONS response from node -1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=0, headerVersion=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=15), ApiVersion(apiKey=2, minVersion=0, maxVersion=8), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=4, minVersion=0, maxVersion=7), ApiVersion(apiKey=5, minVersion=0, maxVersion=4), ApiVersion(apiKey=6, minVersion=0, maxVersion=8), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=8), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=4), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=3), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=58, minVersion=0, maxVersion=0), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=67, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[], zkMigrationReady=false)
[AdminClient clientId=adminclient-1] Node -1 has finalized features epoch: 0, finalized features: [], supported features: [], ZK migration ready: false, API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 15 [usable: 15], ListOffsets(2): 0 to 8 [usable: 8], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): 0 to 7 [usable: 7], StopReplica(5): 0 to 4 [usable: 4], UpdateMetadata(6): 0 to 8 [usable: 8], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 8 [usable: 8], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 4 [usable: 4], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 3], CreateAcls(30): 0 to 3 [usable: 3], DeleteAcls(31): 0 to 3 [usable: 3], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 4], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 3], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 3], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], DescribeQuorum(55): UNSUPPORTED, AlterPartition(56): 0 to 3 [usable: 3], UpdateFeatures(57): 0 to 1 [usable: 1], Envelope(58): 0 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], UnregisterBroker(64): UNSUPPORTED, DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): 0 [usable: 0], ConsumerGroupHeartbeat(68): UNSUPPORTED).
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0)] at 1706133998742
[AdminClient clientId=adminclient-1] Metadata is not ready: we have not fetched metadata from the bootstrap nodes yet.
[AdminClient clientId=adminclient-1] Unable to assign Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) to a node.
[AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to xx.x.xxx:9092 (id: -1 rack: null). correlationId=1, timeoutMs=29606
[AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-1, correlationId=1, headerVersion=2) and timeout 29606 to node -1: MetadataRequestData(topics=[], allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=100)
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0)] at 1706133998744
[AdminClient clientId=adminclient-1] Metadata is not ready: we have not fetched metadata from the bootstrap nodes yet.
[AdminClient clientId=adminclient-1] Unable to assign Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) to a node.
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=100)
[AdminClient clientId=adminclient-1] Received METADATA response from node -1 for request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-1, correlationId=1, headerVersion=2): MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='xx.x.xxx', port=9092, rack=null), MetadataResponseBroker(nodeId=1, host='xx.x.xxx', port=9092, rack=null)], clusterId='xCDtlxlOT-uxomG0SSnBFQ', controllerId=0, topics=[], clusterAuthorizedOperations=-2147483648)
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 1 response(s)
[AdminClient clientId=adminclient-1] Updating cluster metadata to Cluster(id = xCDtlxlOT-uxomG0SSnBFQ, nodes = [xx.x.xxx:9092 (id: 0 rack: null), xx.x.xxx:9092 (id: 1 rack: null)], partitions = [], controller = xx.x.xxx:9092 (id: 0 rack: null))
[AdminClient clientId=adminclient-1] Call(callName=fetchMetadata, deadlineMs=1706134028348, tries=0, nextAllowedTryMs=0) got response MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='xx.x.xxx', port=9092, rack=null), MetadataResponseBroker(nodeId=1, host='xx.x.xxx', port=9092, rack=null)], clusterId='xCDtlxlOT-uxomG0SSnBFQ', controllerId=0, topics=[], clusterAuthorizedOperations=-2147483648)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0)] at 1706133998769
[AdminClient clientId=adminclient-1] Metadata is ready to use.
[AdminClient clientId=adminclient-1] Found least loaded node xx.x.xxx:9092 (id: 1 rack: null) with no active connection
[AdminClient clientId=adminclient-1] Assigned Call(callName=listTopics, deadlineMs=1706134058350, tries=0, nextAllowedTryMs=0) to node xx.x.xxx:9092 (id: 1 rack: null)
Resolved host xx.x.xxx as xx.x.xxx
[AdminClient clientId=adminclient-1] Initiating connection to node xx.x.xxx:9092 (id: 1 rack: null) using address /xx.x.xxx
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: 1 rack: null). Must delay 9698 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=9698)
Added sensor with name node-1.requests-sent
Registered metric named MetricName [name=request-total, group=admin-client-node-metrics, description=The total number of requests sent, tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=request-rate, group=admin-client-node-metrics, description=The number of requests sent per second, tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=request-size-avg, group=admin-client-node-metrics, description=The average size of requests sent., tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=request-size-max, group=admin-client-node-metrics, description=The maximum size of any request sent., tags={client-id=adminclient-1, node-id=node-1}]
Added sensor with name node-1.bytes-sent
Registered metric named MetricName [name=outgoing-byte-total, group=admin-client-node-metrics, description=The total number of outgoing bytes, tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=outgoing-byte-rate, group=admin-client-node-metrics, description=The number of outgoing bytes per second, tags={client-id=adminclient-1, node-id=node-1}]
Added sensor with name node-1.responses-received
Registered metric named MetricName [name=response-total, group=admin-client-node-metrics, description=The total number of responses received, tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=response-rate, group=admin-client-node-metrics, description=The number of responses received per second, tags={client-id=adminclient-1, node-id=node-1}]
Added sensor with name node-1.bytes-received
Registered metric named MetricName [name=incoming-byte-total, group=admin-client-node-metrics, description=The total number of incoming bytes, tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=incoming-byte-rate, group=admin-client-node-metrics, description=The number of incoming bytes per second, tags={client-id=adminclient-1, node-id=node-1}]
Added sensor with name node-1.latency
Registered metric named MetricName [name=request-latency-avg, group=admin-client-node-metrics, description=, tags={client-id=adminclient-1, node-id=node-1}]
Registered metric named MetricName [name=request-latency-max, group=admin-client-node-metrics, description=, tags={client-id=adminclient-1, node-id=node-1}]
[AdminClient clientId=adminclient-1] Created socket with SO_RCVBUF = 65536, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node 1
[AdminClient clientId=adminclient-1] Completed connection to node 1. Fetching API versions.
[AdminClient clientId=adminclient-1] Initiating API versions fetch from node 1.
[AdminClient clientId=adminclient-1] No version information found when sending API_VERSIONS with correlation id 2 to node 1. Assuming version 3.
[AdminClient clientId=adminclient-1] Sending API_VERSIONS request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2, headerVersion=2) and timeout 3600000 to node 1: ApiVersionsRequestData(clientSoftwareName='apache-kafka-java', clientSoftwareVersion='3.6.1')
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1706133998795
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: 1 rack: null). Must delay 9223372036854775807 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=29974)
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1706133998796
[AdminClient clientId=adminclient-1] Client is not ready to send to xx.x.xxx:9092 (id: 1 rack: null). Must delay 9223372036854775807 ms
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=29973)
[AdminClient clientId=adminclient-1] Received API_VERSIONS response from node 1 for request with header RequestHeader(apiKey=API_VERSIONS, apiVersion=3, clientId=adminclient-1, correlationId=2, headerVersion=2): ApiVersionsResponseData(errorCode=0, apiKeys=[ApiVersion(apiKey=0, minVersion=0, maxVersion=9), ApiVersion(apiKey=1, minVersion=0, maxVersion=15), ApiVersion(apiKey=2, minVersion=0, maxVersion=8), ApiVersion(apiKey=3, minVersion=0, maxVersion=12), ApiVersion(apiKey=4, minVersion=0, maxVersion=7), ApiVersion(apiKey=5, minVersion=0, maxVersion=4), ApiVersion(apiKey=6, minVersion=0, maxVersion=8), ApiVersion(apiKey=7, minVersion=0, maxVersion=3), ApiVersion(apiKey=8, minVersion=0, maxVersion=8), ApiVersion(apiKey=9, minVersion=0, maxVersion=8), ApiVersion(apiKey=10, minVersion=0, maxVersion=4), ApiVersion(apiKey=11, minVersion=0, maxVersion=9), ApiVersion(apiKey=12, minVersion=0, maxVersion=4), ApiVersion(apiKey=13, minVersion=0, maxVersion=5), ApiVersion(apiKey=14, minVersion=0, maxVersion=5), ApiVersion(apiKey=15, minVersion=0, maxVersion=5), ApiVersion(apiKey=16, minVersion=0, maxVersion=4), ApiVersion(apiKey=17, minVersion=0, maxVersion=1), ApiVersion(apiKey=18, minVersion=0, maxVersion=3), ApiVersion(apiKey=19, minVersion=0, maxVersion=7), ApiVersion(apiKey=20, minVersion=0, maxVersion=6), ApiVersion(apiKey=21, minVersion=0, maxVersion=2), ApiVersion(apiKey=22, minVersion=0, maxVersion=4), ApiVersion(apiKey=23, minVersion=0, maxVersion=4), ApiVersion(apiKey=24, minVersion=0, maxVersion=4), ApiVersion(apiKey=25, minVersion=0, maxVersion=3), ApiVersion(apiKey=26, minVersion=0, maxVersion=3), ApiVersion(apiKey=27, minVersion=0, maxVersion=1), ApiVersion(apiKey=28, minVersion=0, maxVersion=3), ApiVersion(apiKey=29, minVersion=0, maxVersion=3), ApiVersion(apiKey=30, minVersion=0, maxVersion=3), ApiVersion(apiKey=31, minVersion=0, maxVersion=3), ApiVersion(apiKey=32, minVersion=0, maxVersion=4), ApiVersion(apiKey=33, minVersion=0, maxVersion=2), ApiVersion(apiKey=34, minVersion=0, maxVersion=2), ApiVersion(apiKey=35, minVersion=0, maxVersion=4), ApiVersion(apiKey=36, minVersion=0, maxVersion=2), ApiVersion(apiKey=37, minVersion=0, maxVersion=3), ApiVersion(apiKey=38, minVersion=0, maxVersion=3), ApiVersion(apiKey=39, minVersion=0, maxVersion=2), ApiVersion(apiKey=40, minVersion=0, maxVersion=2), ApiVersion(apiKey=41, minVersion=0, maxVersion=3), ApiVersion(apiKey=42, minVersion=0, maxVersion=2), ApiVersion(apiKey=43, minVersion=0, maxVersion=2), ApiVersion(apiKey=44, minVersion=0, maxVersion=1), ApiVersion(apiKey=45, minVersion=0, maxVersion=0), ApiVersion(apiKey=46, minVersion=0, maxVersion=0), ApiVersion(apiKey=47, minVersion=0, maxVersion=0), ApiVersion(apiKey=48, minVersion=0, maxVersion=1), ApiVersion(apiKey=49, minVersion=0, maxVersion=1), ApiVersion(apiKey=50, minVersion=0, maxVersion=0), ApiVersion(apiKey=51, minVersion=0, maxVersion=0), ApiVersion(apiKey=56, minVersion=0, maxVersion=3), ApiVersion(apiKey=57, minVersion=0, maxVersion=1), ApiVersion(apiKey=58, minVersion=0, maxVersion=0), ApiVersion(apiKey=60, minVersion=0, maxVersion=0), ApiVersion(apiKey=61, minVersion=0, maxVersion=0), ApiVersion(apiKey=65, minVersion=0, maxVersion=0), ApiVersion(apiKey=66, minVersion=0, maxVersion=0), ApiVersion(apiKey=67, minVersion=0, maxVersion=0)], throttleTimeMs=0, supportedFeatures=[], finalizedFeaturesEpoch=0, finalizedFeatures=[], zkMigrationReady=false)
[AdminClient clientId=adminclient-1] Node 1 has finalized features epoch: 0, finalized features: [], supported features: [], ZK migration ready: false, API versions: (Produce(0): 0 to 9 [usable: 9], Fetch(1): 0 to 15 [usable: 15], ListOffsets(2): 0 to 8 [usable: 8], Metadata(3): 0 to 12 [usable: 12], LeaderAndIsr(4): 0 to 7 [usable: 7], StopReplica(5): 0 to 4 [usable: 4], UpdateMetadata(6): 0 to 8 [usable: 8], ControlledShutdown(7): 0 to 3 [usable: 3], OffsetCommit(8): 0 to 8 [usable: 8], OffsetFetch(9): 0 to 8 [usable: 8], FindCoordinator(10): 0 to 4 [usable: 4], JoinGroup(11): 0 to 9 [usable: 9], Heartbeat(12): 0 to 4 [usable: 4], LeaveGroup(13): 0 to 5 [usable: 5], SyncGroup(14): 0 to 5 [usable: 5], DescribeGroups(15): 0 to 5 [usable: 5], ListGroups(16): 0 to 4 [usable: 4], SaslHandshake(17): 0 to 1 [usable: 1], ApiVersions(18): 0 to 3 [usable: 3], CreateTopics(19): 0 to 7 [usable: 7], DeleteTopics(20): 0 to 6 [usable: 6], DeleteRecords(21): 0 to 2 [usable: 2], InitProducerId(22): 0 to 4 [usable: 4], OffsetForLeaderEpoch(23): 0 to 4 [usable: 4], AddPartitionsToTxn(24): 0 to 4 [usable: 4], AddOffsetsToTxn(25): 0 to 3 [usable: 3], EndTxn(26): 0 to 3 [usable: 3], WriteTxnMarkers(27): 0 to 1 [usable: 1], TxnOffsetCommit(28): 0 to 3 [usable: 3], DescribeAcls(29): 0 to 3 [usable: 3], CreateAcls(30): 0 to 3 [usable: 3], DeleteAcls(31): 0 to 3 [usable: 3], DescribeConfigs(32): 0 to 4 [usable: 4], AlterConfigs(33): 0 to 2 [usable: 2], AlterReplicaLogDirs(34): 0 to 2 [usable: 2], DescribeLogDirs(35): 0 to 4 [usable: 4], SaslAuthenticate(36): 0 to 2 [usable: 2], CreatePartitions(37): 0 to 3 [usable: 3], CreateDelegationToken(38): 0 to 3 [usable: 3], RenewDelegationToken(39): 0 to 2 [usable: 2], ExpireDelegationToken(40): 0 to 2 [usable: 2], DescribeDelegationToken(41): 0 to 3 [usable: 3], DeleteGroups(42): 0 to 2 [usable: 2], ElectLeaders(43): 0 to 2 [usable: 2], IncrementalAlterConfigs(44): 0 to 1 [usable: 1], AlterPartitionReassignments(45): 0 [usable: 0], ListPartitionReassignments(46): 0 [usable: 0], OffsetDelete(47): 0 [usable: 0], DescribeClientQuotas(48): 0 to 1 [usable: 1], AlterClientQuotas(49): 0 to 1 [usable: 1], DescribeUserScramCredentials(50): 0 [usable: 0], AlterUserScramCredentials(51): 0 [usable: 0], DescribeQuorum(55): UNSUPPORTED, AlterPartition(56): 0 to 3 [usable: 3], UpdateFeatures(57): 0 to 1 [usable: 1], Envelope(58): 0 [usable: 0], DescribeCluster(60): 0 [usable: 0], DescribeProducers(61): 0 [usable: 0], UnregisterBroker(64): UNSUPPORTED, DescribeTransactions(65): 0 [usable: 0], ListTransactions(66): 0 [usable: 0], AllocateProducerIds(67): 0 [usable: 0], ConsumerGroupHeartbeat(68): UNSUPPORTED).
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1706133998819
[AdminClient clientId=adminclient-1] Sending MetadataRequestData(topics=null, allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false) to xx.x.xxx:9092 (id: 1 rack: null). correlationId=3, timeoutMs=29950
[AdminClient clientId=adminclient-1] Sending METADATA request with header RequestHeader(apiKey=METADATA, apiVersion=12, clientId=adminclient-1, correlationId=3, headerVersion=2) and timeout 29950 to node 1: MetadataRequestData(topics=null, allowAutoTopicCreation=true, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=59531)
[AdminClient clientId=adminclient-1] KafkaClient#poll retrieved 0 response(s)
[AdminClient clientId=adminclient-1] Trying to choose nodes for [] at 1706133998820
[AdminClient clientId=adminclient-1] Entering KafkaClient#poll(timeout=59530)

so any help on that or something i could discover more ?