Answers: <p>Things we have checked were what happens with the background io queue and the buffer. It turned out our producers were ok as lag was increasing, but as offset was static on check of consumer handlers it turned out we don't send acknowledgement to Kafka on read, leading to Kafka sending the same message after timeout on each attempt to consume. On java code side we have just made sure to send ack.</p>
  
Answers: CC BY-SA 4.0
Answers: 2020-06-16T07:29:30.403Z
Answers: 2020-06-16T07:29:30.403Z
Answers: anVzdGFub3RoZXJodW1hbg
Body: <p>We are using Confluent Kafka cluster on production K8s.
  It looks like our consumers are stuck in a loop as offset is static and lag keeps going up.
  When I restart zookeper/brokers (persistent volume) the offset moves by a small amount and then it is stuck again. I have checked my consumers and only 1 of them has assigned something, that is processed and all assignments finish after. What happens on consumer side is that sometimes we have an error, we tried ignoring the message but Kafka is still affected. What can be done outside of the default config. We are not sure why message are stopped being consumed or it looks like there is some loop state leading to attempting to process same message over and over agian.</p>
  
Tags: <kafka>
Title: Confluent Kafka on K8s - offset is stuck, lag increases, what can be done?
UserDisplayName: anVzdGFub3RoZXJodW1hbg
ThreadText: # Confluent Kafka on K8s - offset is stuck, lag increases, what can be done?
  
  (Asked by: username_0 on 2020-06-10)
  
  We are using Confluent Kafka cluster on production K8s.
  It looks like our consumers are stuck in a loop as offset is static and lag keeps going up.
  When I restart zookeper/brokers (persistent volume) the offset moves by a small amount and then it is stuck again. I have checked my consumers and only 1 of them has assigned something, that is processed and all assignments finish after. What happens on consumer side is that sometimes we have an error, we tried ignoring the message but Kafka is still affected. What can be done outside of the default config. We are not sure why message are stopped being consumed or it looks like there is some loop state leading to attempting to process same message over and over agian.
  
  **Tags:** kafka
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-06-16. Score: 0 | Accepted answer
  
  Things we have checked were what happens with the background io queue and the buffer. It turned out our producers were ok as lag was increasing, but as offset was static on check of consumer handlers it turned out we don't send acknowledgement to Kafka on read, leading to Kafka sending the same message after timeout on each attempt to consume. On java code side we have just made sure to send ack.
  
  
  ---
  
  