KafkaIndexSupervisor becomes UNHEALTHY, blocking partition consumption #18779
Description
Druid Version
24.0.1
Supervisor Type
KafkaIndexSupervisor
Problem Description
We observed that when a Kafka ingestion task's duration exceeds the configured segmentGranularity, the Supervisor enters the UNHEALTHY_SUPERVISOR state. As a result, certain Kafka partitions stop being consumed, even if the partition contains a very small amount of data (only a few thousand rows).
This occurs even under minimal load, which indicates that the issue is likely related to Supervisor's internal task handling rather than Kafka volume or resource exhaustion.
Additionally, the logs sometimes show repeated exceptions like:
java.lang.IllegalStateException: Optional.get() cannot be called on an absent value