cannot fully disable kafka binder and cloud stack for testing

[JohnnyXavier]

hi,
we are using

      <dependency>
        <groupId>org.springframework.cloud</groupId>
        <artifactId>spring-cloud-starter-stream-kafka</artifactId>
        <version>${spring-cloud-stream-binder-kafka.version}</version> //3.1.3
      </dependency>

When running tests the AdminClient (and the larger cloud stack) attempts to init on every test.
We do have an application.yaml with a kafka binder configured which is the one tests are trying to connect to.

The AdminClient tries to connect to a broker that does not exist, consequently failing to create consumers, admin client, etc and adding a lot of time until finally timing out.

Given that not all tests are about testing broker behavior this causes a lot of delay, mostly locally when trying to unit test certain behavior unrelated to brokers.

I tried to exclude kafka and cloud autoconfigurations without success
these are the ones I took out starting with KafkaAutoConfiguration alone and then adding the rest 1 by 1.

  autoconfigure:
    exclude:
      - org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration
      - org.springframework.boot.actuate.autoconfigure.metrics.KafkaMetricsAutoConfiguration
      - org.springframework.cloud.stream.config.BindingsEndpointAutoConfiguration
      - org.springframework.cloud.stream.config.ChannelBindingAutoConfiguration
      - org.springframework.cloud.stream.config.ChannelsEndpointAutoConfiguration
      - org.springframework.cloud.stream.config.BindersHealthIndicatorAutoConfiguration

also tried excluding them via code:

@SpringBootApplication( exclude = {KafkaAutoConfiguration.class, .........})

even with all those out the KafkaAutoconfiguration keeps getting beaned in as well as KafkaBinderConfiguration

I also tried to exclude non autoconfigurations beans as KafkaBinderConfiguration, KafkaBinderHealthIndicatorConfiguration, DefaultBinderFactory, without success.

I could not find a way to fully disable the cloud stack as some dependency keeps pulling Kafka and binder back in and KafkaAutoconfiguration keeps getting called

---

I finally "hacked" a solution by making the attempt to connect fail fast like this

  cloud:
    stream:
      kafka:
        binder:
          brokers:

which makes the AdminClient to fail immediately on an empty broker and we don't have to wait for the cloud stack to timeout.

2021-06-11 08:59:34,114 [main] INFO  org.springframework.cloud.stream.binder.DefaultBinderFactory - Creating binder: kafka
2021-06-11 08:59:34,329 [main] INFO  org.springframework.cloud.stream.binder.DefaultBinderFactory - Caching the binder: kafka
2021-06-11 08:59:34,330 [main] INFO  org.springframework.cloud.stream.binder.DefaultBinderFactory - Retrieving cached binder: kafka
2021-06-11 08:59:34,408 [main] INFO  org.apache.kafka.clients.admin.AdminClientConfig - AdminClientConfig values: 
	bootstrap.servers = []

Seems there should be a way to completely exclude the cloud stack so it won't load any beans related to binders, channels, kafka, etc

the first one I can see gets called is

2021-06-11 09:07:36,266 [main] INFO  org.springframework.integration.config.DefaultConfiguringBeanFactoryPostProcessor - No bean named 'errorChannel' has been explicitly defined. Therefore, a default PublishSubscribeChannel will be created.

Can u advise if this is by design?

Is it possible to include a "CloudAutoconfiguration" bean that will be at the top of init of the rest of the stack so we can disable it completely maybe like this:?

  cloud:
      disabled: true

---

addendum:

adding this test dep

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream-test-support</artifactId>
            <scope>test</scope>
        </dependency>

eliminates the AdminClient connecting and solves the timeout delay problem when testing non broker related beans, thou still pulls cloud dependencies, add test channels, etc

that solution thou, required us to exclude the TestSupportBinderAutoConfiguration.class bean to test broker with testContainers on our broker related tests...

Thanks a lot.

[garyrussell]

Have you tried setting the binding's autoStartup property to false?

https://docs.spring.io/spring-cloud-stream/docs/3.1.3/reference/html/spring-cloud-stream.html#_consumer_properties

https://docs.spring.io/spring-cloud-stream/docs/3.1.3/reference/html/spring-cloud-stream.html#_producer_properties

[sobychacko]

Another thing to try if you don't want to connect to Kafka in the tests, you can use the new test binder dependency from Spring Cloud Stream.

 <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream</artifactId>
            <type>test-jar</type>
            <classifier>test-binder</classifier>
            <scope>test</scope>
        </dependency>

and then use its input and output destination capabilities to test.

[JohnnyXavier]

Hi Gary and Soby,

thanks for getting back so quickly!

@garyrussell: unfortunately those configs did not make a difference
@sobychacko: we do want to connect to kafka on tests but not to all of them all the time. Probably under 10% of our tests hit kafka behavior

the idea would be to be able to turn cloud functionality on or off with a property or simple exclusion, that way for the tests that don't have a kafka requirement, we won't worry about the connection attempts.

Being able to disable the binder would also be very useful when running locally as the broker does not add much functionality to our particular app in local environments

so being able to disable the binder on demand would be fantastic both for tests and for running locally.

---

I did something thou that can give u a better hint on how to reproduce this.

from the spring started select lombok webflux and cloud kafka and on the pom use

        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream-binder-kafka-streams</artifactId>
        </dependency>

instead of the included cloud one

remove the test dependencies

then just add to the application.java the bean from the documentation

    @Bean
    public Consumer<KStream<Object, String>> process() {
        return input -> input.foreach((key, value) -> {
            System.out.println("Key: " + key + " Value: " + value);
        });
    }

run the app or the tests and try to:
prevent the adminclient to try to connect when running on local
be able to run 1 non kafka related test without the binder running and 1 kafka dependent test

with just that you can reproduce the mentioned behavior

with version 2.5.1 of boot I could disable the connection attempts with the following exclusions

@SpringBootApplication(exclude = {KafkaAutoConfiguration.class, KafkaStreamsBinderSupportAutoConfiguration.class,
        KafkaStreamsTopologyEndpointAutoConfiguration.class, KafkaStreamsFunctionAutoConfiguration.class})

this has the same effect

spring:
  autoconfigure:
    exclude:
      - org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration
      - org.springframework.cloud.stream.binder.kafka.streams.KafkaStreamsBinderSupportAutoConfiguration
      - org.springframework.cloud.stream.binder.kafka.streams.endpoint.KafkaStreamsTopologyEndpointAutoConfiguration
      - org.springframework.cloud.stream.binder.kafka.streams.function.KafkaStreamsFunctionAutoConfiguration

maybe with spring boot 2.5.1 and the above exclusions, although there are still logs about channels and binders, we don't have the admin client trying to connect continuously.

hope this helps

the idea basically is to be able to disable the behavior with props so we can easily switch profiles when running locally or during tests

thanks again so much for your time !!!!

[sobychacko]

@JohnnyXavier Thanks for the explanation above. Any chance you can set this up in a small project and share it as a GH repository? Please include all the dependencies you need (and the exclusions) to make it work. That way, it is easier for us to reproduce this on our end and see it in action. Once we identify the root cause, then it would be easier to fix. Thank you!

[JohnnyXavier]

hey @sobychacko sure!

https://github.com/JohnnyXavier/springboot_1089_kafka_binder.git

it is a very simple example.
without locally running kafka

• try running run the app without specifying a profile.
  • You should see the adminClient attempting to reconnect at intervals indefinitely.
• try running the app with workaround as active profile.
  • You should see the adminClient fail and never attempt to reconnect.

Thanks a lot again!

[sobychacko]

@JohnnyXavier when running this app, you certainly will see that exception, as the app tries to connect to a Kafka broker. I thought your original issue was with running tests in which you don't need to connect to the Kafka broker. Where is the disconnect?

One other thing that you may want to consider is the use of EmbeddedKafka for testing. This way, your test profile has access to a real Kafka cluster and then you can also have tests that don't require Kafka as well. Look at how these sample projects are structured.

https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/kafka-streams-samples/kafka-streams-word-count

https://github.com/spring-cloud/spring-cloud-stream-samples/tree/main/kafka-streams-samples/kafka-streams-inventory-count

[JohnnyXavier]

Hey @sobychacko, thanks for the above!

Testing is solved yes! thanks very much for your input.
On my second comment the follow up note was:

Being able to disable the binder would also be very useful when running locally as the broker does not add much functionality to our particular app in local environments

so being able to disable the binder on demand would be fantastic both for tests and for running locally.

Is there a way we can avoid the cloud/binder/kafka stack from running altogether somehow? If I can switch profiles, and on a application-local profile we could prevent the kafka stack to even running would be fantastic.

thanks a lot!