Question: we are having problem with Kafka. sometimes Suddenly, without warning we go out of Synchronization and start to get exceptions when emitting events.

the exception we are getting is 

```
java.io.IOException: Too many open files

```

it seems this is a generic exception thrown by Kafka in many cases. We investigated it a little and we think the root cause is when trying to emit events to some topic, it fails because kafka dosen't have a leader partition for this topic

can someone help ?

Answer: 
I assume that you are on Linux. If that is the case, then what's happening is that you are running out of open file descriptors. The real question is why this is happening.

Linux by default generally keeps this number fairly low. You can check the actual value via ulimit:

```
ulimit -a | grep "open files"

```

You can then set that value via, again ulimit:

```
sudo ulimit -n 4096

```

That said, unless the Kafka host in question has lots of topics / partitions it is unusual to hit that limit. What's probably happening is that some other process is keeping files or connections open. In order to figure out which process you're going to have to do some detective work with lsof.
In our case our Kafka topics were accidentally configured `"segment.ms" = 20000` and were generating new log segments every 20 seconds when the default is 604800000 (1 week).

We are using amazon's msk, so we didn't have the ability ourselves to run the commands, however amazon support was able to monitor it for us. That caused this issue, but then some of the nodes were not recovering.

We took two steps..

**1) Force Compaction**

We set the retention and ratio low to clean up

```
"delete.retention.ms" = 100
"min.cleanable.dirty.ratio" = "0.01"

```

One of the nodes, was able to recover... but another did not seem to recover to the point where Kafka would actually run compaction, it seemed to be the "leader" on one of the largest topics.

**2) Free up space**

We decided to destroy the large topic in the hopes it would unblock the node. Eventually the compaction seemed to run on all the nodes.

Later we restored the topic that we destroyed with the new segmentation settings, and have been running fine since.