Schema Registry is continue working even if KafkaStoreReaderThread has been stopped unexpectedly

[drozd0]

Hello,

I would raise a question about decision behind keeping Schema Registry healthy when KafkaStoreReaderThread died.

We have a case when Kafka Access Control List(ACLs) was applied and KafkaStoreReaderThread has been stopped due to exception KafkaStoreReader thread has died for an unknown reason

After that Schema Registry may work only in readonly mode with limited state which has been read before the thread crash.

As I understand, only Schema Registry restart may launch the KafkaStoreReaderThread again. If it's true then how to know that the KafkaStoreReaderThread has been stopped to restart the service(k8s pod) again?

We have an health check for /subjects endpoint, I also tried /metadata endpoint and k8s tcp health check on Schema Registry port.
But Schema Registry still returns OK response:)

To summarize:
Do we really need keep Schema Registry working if KafkaStoreReaderThread died? Or kill the Schema Registry itself to let system know about the failure?

As a suggestion, maybe it's better to kill Schema Registry by throwing FatalExitError here?

What do you think?

Thank you!

[drozd0]

@rayokota do you have any thoughts? Thank you!