[FanOutRecordsPublisher] ClosedChannelException WARN logs with Netty HTTP2 client (KCL 3.1.x)

[anjeongkyun]

Summary

When running with Spring Cloud Stream Kinesis Binder (4.0.4) using KCL 3.1.x under JDK 21, I observed repeated ClosedChannelException warnings caused by Netty HTTP2 channel closures.
KCL recovers by recreating subscriptions, but the WARN logs keep repeating, creating operational noise.

Logs (excerpt)

2025-09-21 14:44:04 WARN s.a.k.r.f.FanOutRecordsPublisher:350 - shardId-000000000003: [SubscriptionLifetime] - (FanOutRecordsPublisher#errorOccurred) ...
java.io.IOException: An error occurred on the connection: java.nio.channels.ClosedChannelException
at software.amazon.awssdk.http.nio.netty.internal.http2.MultiplexedChannelRecord.decorateConnectionException(...)
...
2025-09-21 14:44:05 WARN s.a.k.l.ShardConsumerSubscriber:115 - shardId-000000000003: Failure occurred in retrieval. Restarting data requests

Environment

• Spring Boot: 3.3.5
• Spring Cloud Stream Binder for Kinesis: 4.0.4
• Spring Integration AWS: 3.0.8
• Amazon Kinesis Client (KCL): 3.1.x (pulled transitively by the binder)
• JDK: 21
• Fan-Out: enabled

KinesisAsyncClient Bean

@Bean
@Primary
open fun kinesisClient(): KinesisAsyncClient {
    val httpClient =
        NettyNioAsyncHttpClient.builder()
            .protocol(Protocol.HTTP2)
            .http2Configuration { it.healthCheckPingPeriod(Duration.ofSeconds(60)) }
            .connectionMaxIdleTime(Duration.ofSeconds(300))
            .build()

    return KinesisAsyncClient.builder()
        .httpClient(httpClient)
        .region(Region.of(region))
        .apply { endpoint?.let { endpointOverride(URI.create(it)) } }
        .build()
}

Spring Cloud Stream configuration (excerpt)

spring:
  cloud:
    stream:
      bindings:
        handlePurchaseDomainEvent-in-0:
          group: shared-purchase-event-consumer
          destination: shared-purchase-domain-event-stream
          consumer:
            concurrency: 4
        handlePrepaidCardDomainEvent-in-0:
          group: shared-prepaid-card-event-consumer
          destination: shared-prepaid-card-domain-event-stream
          consumer:
            concurrency: 4
      kinesis:
        binder:
          auto-create-stream: false
          kpl-kcl-enabled: true
          checkpoint:
            table: sharedPurchaseIntegrationMetadataStore
          locks:
            table: sharedPurchaseIntegrationLockRegistry
          enable-observation: false
        bindings:
          handlePurchaseDomainEvent:
            consumer:
              fan-out: true
              metrics-level: NONE
              checkpointMode: batch
              idle-between-polls: 250
              consumerBackoff: 100
              gracefulShutdownTimeout: 20000
          handlePrepaidCardDomainEvent:
            consumer:
              fan-out: true
              metrics-level: NONE
              checkpointMode: batch
              idle-between-polls: 250
              consumerBackoff: 100
              gracefulShutdownTimeout: 20000

Observed Behavior

WARN logs for ClosedChannelException repeat frequently when using Fan-Out consumers.
KCL retries and continues consuming, but the WARN noise persists.

Expected Behavior

Channel closures should be handled more gracefully.
Either recover silently, or downgrade the log level to INFO/DEBUG unless data loss occurs.

Impact

No obvious data loss, but WARN spam makes logs noisy.
Operational monitoring systems treat this as an error, making it difficult to distinguish critical issues.

Additional Context

This appears to be triggered when Http2MultiplexedChannelPool closes parent channels, propagating exceptions to all child channels.
Might be related to idle connection management in Netty + AWS SDK v2.

Questions

• Is this behavior expected in the current design of FanOutRecordsPublisher / ShardConsumerSubscriber?
  I would like to better understand which part of the connection handling logic is responsible for repeatedly logging these ClosedChannelExceptions.
• If this is considered an improvement opportunity (for example, adjusting the log level or handling channel closure more gracefully), I am interested in contributing a fix.
  Could you please advise whether such a contribution would be welcome, and which area of the codebase would be most relevant to look into?

[anjeongkyun]

I did some further digging into the code path behind these WARN logs and wanted to share my findings.

From what I can see, the sequence is:

1. FanOutRecordsPublisher#errorOccurred is invoked when the Netty HTTP2 channel closes (e.g. ClosedChannelException).
2. This immediately calls subscriber.onError(t).
3. In ShardConsumerSubscriber#onError, the subscription is cancelled and the consumer is marked as FAILED.
4. The Scheduler then detects the FAILED state and restarts the ShardConsumer after a fixed backoff (consumerBackoffMillis).

So the WARN messages we observe in this issue are tied to this onError → FAILED → restart cycle.
In my environment it only happens once per channel closure, but if closures happen frequently the WARNs could appear repeatedly, creating operational noise.

Currently, this recovery loop uses only a fixed backoff. There is no exponential backoff or jitter applied at the KCL layer (aside from what the AWS SDK retry policy does under the hood).
This makes the retry behavior somewhat inconsistent: sometimes SDK-level exponential backoff, sometimes KCL-level fixed backoff, sometimes immediate restart.

I wonder if it would make sense to introduce a configurable RetryPolicy at the KCL level.
That could allow operators to apply exponential backoff + jitter (similar to what Kafka clients and other systems do), reduce WARN spam when transient failures occur, and make retry handling more predictable overall.

Would this kind of contribution be considered useful for the project? If so, I’d be happy to explore it and see where in the codebase such a policy would best be plugged in.