title: "Connection reset by peer" and "Too many open files" exceptions wile trying to scale out Vert.x server
body: <p>I am trying to test the scalability of a Java Vert.x webserver (which is based on Netty) on Mac OS X. For that, I have increased the connection limit to 1 million:</p>
  
  <pre><code>sudo sysctl -w kern.maxfiles=1000200
  sudo sysctl -w kern.maxfilesperproc=1000100
  sudo ulimit -n 1000000
  </code></pre>
  
  <p>Then I have created localhost aliases with:</p>
  
  <pre><code>for i in `seq 200 250`; do sudo ifconfig lo0 alias 172.16.123.$i ; done
  </code></pre>
  
  <p>I am now able to connect to these IP addresses; however, after around 1000-2000 established connections I get "Connection reset by peer" connections:</p>
  
  <pre><code>java.net.SocketException: Connection reset by peer
      at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
      at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
      at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:191)
      at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:228)
      at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:497)
      at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:447)
  </code></pre>
  
  <p>They are followed by "Too many open files" exceptions:</p>
  
  <pre><code>Caused by: java.net.SocketException: Too many open files
      at sun.nio.ch.Net.socket0(Native Method)
      at sun.nio.ch.Net.socket(Net.java:393)
      at sun.nio.ch.Net.socket(Net.java:386)
      at sun.nio.ch.SocketChannelImpl.&lt;init&gt;(SocketChannelImpl.java:104)
      at sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
      at java.nio.channels.SocketChannel.open(SocketChannel.java:145)
      at io.netty.channel.socket.nio.NioSocketChannel.newSocket(NioSocketChannel.java:48)
  </code></pre>
  
  <p>Interestingly, when I list all connections with netstat then I see that there are a lot of connections in the FIN_WAIT_1 state:</p>
  
  <pre><code>tcp4       0    213  172.16.123.230.55618   172.16.123.230.8877    FIN_WAIT_1 
  tcp4       0    213  172.16.123.229.55624   172.16.123.229.8877    FIN_WAIT_1 
  tcp4       0    213  172.16.123.228.55617   172.16.123.228.8877    FIN_WAIT_1 
  tcp4       0    213  172.16.123.227.55616   172.16.123.227.8877    FIN_WAIT_1 
  tcp4       0    213  172.16.123.226.55612   172.16.123.226.8877    FIN_WAIT_1 
  </code></pre>
  
  <p>Other results:</p>
  
  <pre><code>$ netstat -a | wc -l:
  14282
  $ lsof | wc -l
  6922
  </code></pre>
  
  <p>Server code (simplified):</p>
  
  <pre><code>HttpServer server = vertx.createHttpServer();
  server.websocketHandler(ws -&gt; {
      connections++;
      ws.exceptionHandler(t -&gt; t.printStackTrace());
      ws.closeHandler(h -&gt; {
          connections--;
      });
  
      if (ws.path().equals("/app")) {
          ws.dataHandler(data -&gt; {
              // ...
          });
      } else {
          ws.reject();
      }
  }).listen(8877);
  </code></pre>
  
  <p>Client code (simplified):</p>
  
  <pre><code>Handler&lt;Void&gt; connectHandler = new Handler&lt;Void&gt;() {
      @Override
      public void handle(Void e) {
          HttpClient client = vertx.createHttpClient().setHost(nextHost()).setPort(8877);
  
          client.connectWebsocket("/app", ws -&gt; {
              connections++;
              ws.exceptionHandler(t -&gt; t.printStackTrace());
              ws.closeHandler(h -&gt; {
                  connections--;
              });
  
              // ...
          });
  
          if (connections &lt; 1000000) {
              vertx.runOnContext(this);
          }
      }
  };
  connectHandler.handle(null);
  </code></pre>
  
  <p>Both started with (on the same computer):</p>
  
  <pre><code>vertx run Class -cp classes:... -instances 8
  </code></pre>
  
  <p>Does anybody have an idea what could be the problem? I am not yet experienced with such setups and maybe I have just made a simple mistake. It doesn't help to increase the accept backlog.</p>
  
  <p>Further information:</p>
  
  <ul>
  <li><a href="http://krypted.com/mac-os-x/maximum-files-in-mac-os-x/" rel="noreferrer">http://krypted.com/mac-os-x/maximum-files-in-mac-os-x/</a></li>
  <li><a href="https://stackoverflow.com/questions/87442/virtual-network-interface-in-mac-os-x">Virtual network interface in Mac OS X</a></li>
  </ul>
  
  <p><strong>Update:</strong></p>
  
  <p>After adding </p>
  
  <pre><code>limit maxfiles 1000100 2000200
  </code></pre>
  
  <p>to /etc/launchd.conf and </p>
  
  <pre><code>kern.maxfiles=2000400
  kern.maxfilesperproc=1000200
  </code></pre>
  
  <p>to /etc/sysctl.conf and restarting the computer, it has gotten already a little bit better. However, after I while I get now "Too many open files" exceptions even for reading normal files although there are not more than 10000 open files:</p>
  
  <pre><code>Caused by: sun.nio.fs.UnixException: Too many open files
          at sun.nio.fs.UnixNativeDispatcher.getcwd(Native Method)
          at sun.nio.fs.UnixFileSystem.&lt;init&gt;(UnixFileSystem.java:67)
          at sun.nio.fs.BsdFileSystem.&lt;init&gt;(BsdFileSystem.java:41)
          at sun.nio.fs.MacOSXFileSystem.&lt;init&gt;(MacOSXFileSystem.java:44)
          at sun.nio.fs.MacOSXFileSystemProvider.newFileSystem(MacOSXFileSystemProvider.java:45)
          at sun.nio.fs.MacOSXFileSystemProvider.newFileSystem(MacOSXFileSystemProvider.java:38)
          at sun.nio.fs.UnixFileSystemProvider.&lt;init&gt;(UnixFileSystemProvider.java:56)
          at sun.nio.fs.BsdFileSystemProvider.&lt;init&gt;(BsdFileSystemProvider.java:36)
          at sun.nio.fs.MacOSXFileSystemProvider.&lt;init&gt;(MacOSXFileSystemProvider.java:40)
  </code></pre>
  
  <p><strong>Update 2</strong>:</p>
  
  <p>I have tried several other options as well but it seems like there is a limit at exactly 10180 connections which cannot be exceeded. Unless somebody has another idea, I will probably just switch to linux.</p>
  
  <p><strong>Update 3</strong>:</p>
  
  <p>After adding the JVM option "-XX:-MaxFDLimit" I can now open 16331 connections until I get the following exceptions:</p>
  
  <pre><code>java.net.BindException: Can't assign requested address
          at sun.nio.ch.Net.connect0(Native Method)
          at sun.nio.ch.Net.connect(Net.java:435)
          at sun.nio.ch.Net.connect(Net.java:427)
          at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:643)
          at io.netty.channel.socket.nio.NioSocketChannel.doConnect(NioSocketChannel.java:176)
  </code></pre>
  
  <p>This is probably because of the default portrange (16383 ports):</p>
  
  <pre><code>net.inet.ip.portrange.first: 49152
  net.inet.ip.portrange.last: 65535
  </code></pre>
  
  <p>However, since I use 50 localhost alises I would have expected to be able to open up to 16383 * 50 = 819150 connections. Or is it a known limitation that you cannot use the same port for different aliases?</p>
  
  <p><strong>Update 4</strong>:</p>
  
  <p>I have tried to create virtual interfaces with networksetup but the result is the same:</p>
  
  <pre><code>for i in `seq 200 250`; do sudo networksetup -createnetworkservice lo$i lo0 ; sudo networksetup -setmanual lo$i 172.16.123.$i 255.240.0.0 ; done
  </code></pre>