title: kafka + how to avoid running out of disk storage
body: <p>I want to described the following case that was on one of our production cluster</p>
  
  <p>We have ambari cluster with HDP version 2.6.4</p>
  
  <p>Cluster include 3 kafka machines – while each kafka have disk with 5 T</p>
  
  <p>What we saw is that all kafka disks was with 100% size , so kafka disk was full and this is the reason that all kafka brokers was failed</p>
  
  <pre><code>df -h /kafka
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sdb         5T   5T   23M   100% /var/kafka
  </code></pre>
  
  <p>After investigation we saw that <code>log.retention.hours=7 days</code></p>
  
  <p>So seems that purging is after 7 days and maybe this is the reason that kafka disks are full with 100% even if they are huge – 5T</p>
  
  <p>What we want to do now – is how to avoid this case in the future?</p>
  
  <p>So</p>
  
  <p>We want to know – how to avoid full used capacity on kafka disks</p>
  
  <p>What we need to set in Kafka config in order to purge the kafka disk according to the disk size – is it possible ?</p>
  
  <p>And how to know the right value of <code>log.retention.hours</code> ? according to the disk size or other?</p>