Skip to content

KAFKA-20118: Improve the error message caused by normally moving replica#21470

Open
Parkerhiphop wants to merge 1 commit intoapache:trunkfrom
Parkerhiphop:KAFKA-20118-improve-error-message
Open

KAFKA-20118: Improve the error message caused by normally moving replica#21470
Parkerhiphop wants to merge 1 commit intoapache:trunkfrom
Parkerhiphop:KAFKA-20118-improve-error-message

Conversation

@Parkerhiphop
Copy link
Contributor

@Parkerhiphop Parkerhiphop commented Feb 13, 2026

Background

Currently, when a broker receives a LeaderAndIsrRequest for a
partition it does not yet host—most commonly during a partition
reassignment or new topic creation—it logs a high-severity ERROR
message:

ERROR [Broker id=$brokerId] Expected partition $tp with topic id $topicId to exist, but it was missing. Creating...

This message is misleading to operators as it implies a state of
inconsistency or data loss ("missing," "expected to exist"), when in
fact the behavior is a normal part of the partition lifecycle. During a
move, the destination broker is expected not to have the partition
locally until the transition begins.

Changes:

  1. Downgraded the log level from ERROR to INFO.
  2. Updated the message string to focus on the action ("initializing")
    rather than the "missing" state.
  3. The new message is meant to neutral and accurately describes both
    partition reassignment and initial topic creation scenarios.

Impact:

Reduces "log noise" and prevents false-positive alerts in monitoring
systems that trigger on ERROR strings in the state.change.logger.

How to Reproduce

This scenario simulates a standard Cluster Expansion or Partition
Reassignment.

  1. Setup a 2-node KRaft ClusterConfigure two brokers ($ID=1, 2$) to
    share the same controller quorum.
CLUSTER_ID=$(./bin/kafka-storage.sh random-uuid)
./bin/kafka-storage.sh format -t $CLUSTER_ID -c
config/server1.properties
./bin/kafka-storage.sh format -t $CLUSTER_ID -c
config/server2.properties

LOG_DIR=/tmp/kafka-logs-1-logs ./bin/kafka-server-start.sh -daemon
config/server1.properties  LOG_DIR=/tmp/kafka-logs-2-logs
./bin/kafka-server-start.sh -daemon config/server2.properties

  1. Create a Topic on Broker 1

bin/kafka-topics.sh --create --topic reproduction-test --bootstrap-server localhost:9092 --replica-assignment 1

Run a command to create a topic explicitly assigned to Broker 1. At
this point, Broker 2 knows nothing about this partition.

  1. Perform a Normal Reassignment to Broker 2

./bin/kafka-reassign-partitions.sh --bootstrap-server localhost:9092 --reassignment-json-file move.json --execute

{
  "version": 1,
  "partitions": [{ "topic": "reproduction-test", "partition": 0,
"replicas": [2] }]
}
  1. Observed Logs on Destination Broker (Broker 2)

cat /tmp/kafka-logs-2-logs/state-change.log

❌ BEFORE (Current Misleading ERROR)
The ERROR level and the word "missing" falsely imply a system failure
during a routine move.

[2026-02-15 21:44:41,885] INFO [Broker id=2] Transitioning 1
partition(s) to local followers. (state.change.logger)
[2026-02-15 21:44:41,889] ERROR [Broker id=2] Expected partition
reproduction-test-0 with topic id QPz7oe_tQ-mhDdXQ1cb8fQ to exist, but
it was missing. Creating... (state.change.logger)
[2026-02-15 21:44:41,907] INFO [Broker id=2] Follower
reproduction-test-0 starts at leader epoch 0 from offset 0...

✅ AFTER (This PR)
The log is now a neutral INFO and accurately describes the
initialization of a new replica, consistent with the surrounding
lifecycle logs.

[2026-02-15 21:44:41,885] INFO [Broker id=2] Transitioning 1
partition(s) to local followers. (state.change.logger)
[2026-02-15 21:44:41,889] INFO [Broker id=2] Partition
reproduction-test-0 (ID: QPz7oe_tQ-mhDdXQ1cb8fQ) not found locally;
initializing new replica... (state.change.logger)
[2026-02-15 21:44:41,907] INFO [Broker id=2] Follower
reproduction-test-0 starts at leader epoch 0 from offset 0...

@github-actions github-actions bot added triage PRs from the community core Kafka Broker small Small PRs labels Feb 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-approved core Kafka Broker small Small PRs triage PRs from the community

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants