title: Cassandra com.datastax.driver.core.exceptions.NoHostAvailableException
body: <p>I have been struggling to make Cassandra work. It works fine with small tests but as soon as I scale it works for a while but then fails. Loading of the data is not problem but it fails when the database is queried. As said it works well for a little while and afterwards it displays the following message.</p>
  
  <pre><code>com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: xxxx/xx.xx.xx.xx:xx (com.datastax.driver.core.exceptions.TransportException: [xxxx/xx.xx.xx.xx:xx] Connection has been closed), xxxx/xx.xx.xx.xx:xx (com.datastax.driver.core.exceptions.TransportException: [xxxx/xx.xx.xx.xx:xx] Connection has been closed), xxxx/xx.xx.xx.xx:xx (com.datastax.driver.core.exceptions.TransportException: [xxxx/xx.xx.xx.xx:xx] Connection has been closed), xxxx/xx.xx.xx.xx:xx [only showing errors of first 3 hosts, use getErrors() for more details])
  
  All host(s) tried for query failed (tried: xxxx/xx.xx.xx.xx:xx (com.datastax.driver.core.exceptions.TransportException: [xxxx/xx.xx.xx.xx:xx] Connection has been closed), xxxx/xx.xx.xx.xx:xx (com.datastax.driver.core.exceptions.TransportException: [xxxx/xx.xx.xx.xx:xx] Connection has been closed), xxxx/xx.xx.xx.xx:xx (com.datastax.driver.core.exceptions.TransportException: [xxxx/xx.xx.xx.xx:xx] Connection has been closed),  [only showing errors of first 3 hosts, use getErrors() for more details])
  </code></pre>
  
  <p><strong>VERSION</strong></p>
  
  <p>Cassandra 3.0, using java driver 3.0.0 </p>
  
  <p><strong>GENERAL SETTINGS</strong></p>
  
  <p>4 exclusive local linux clusters for testing with default MAX_HEAP_MEMORY</p>
  
  <p><strong>SESSION AND CLUSTER CREATION</strong></p>
  
  <pre><code>LoadBalancingPolicy loadBalancingPolicy = new RoundRobinPolicy();
  
  SocketOptions socketOptions = new SocketOptions();
  socketOptions.setConnectTimeoutMillis(900000000);
  socketOptions.setReadTimeoutMillis(900000000);
  
  PoolingOptions poolingOptions = new PoolingOptions();
  poolingOptions
        .setMaxRequestsPerConnection(HostDistance.LOCAL, 32768)
        .setMaxRequestsPerConnection(HostDistance.REMOTE, 2000)
        .setCoreConnectionsPerHost(HostDistance.LOCAL, 1000)
        .setCoreConnectionsPerHost(HostDistance.REMOTE, 1000)
        .setMaxConnectionsPerHost(HostDistance.LOCAL, 1000)
        .setMaxConnectionsPerHost(HostDistance.REMOTE, 1000)
        .setNewConnectionThreshold(HostDistance.LOCAL, 128)
        .setNewConnectionThreshold(HostDistance.REMOTE, 128);
  
  cluster = Cluster.builder()
        .addContactPointsWithPorts(addresses)
        .withLoadBalancingPolicy(loadBalancingPolicy)
        .withQueryOptions(new QueryOptions().setFetchSize(100000))
        .withSocketOptions(socketOptions)
        .withPoolingOptions(poolingOptions)
        .build();
  
  session = cluster.connect();
  </code></pre>
  
  <p><strong>TABLE CREATION</strong> -> This works fine tables are properly created and loaded</p>
  
  <pre><code>session.execute("CREATE KEYSPACE IF NOT EXISTS KEYSPACE_NAME WITH replication = {'class':'SimpleStrategy', 'replication_factor': 4};");
  session.execute("USE KEYSPACE_NAME;");
  String query =
          "CREATE TABLE IF NOT EXISTS KEYSPACE_NAME.TABLE_NAME 
          ( key blob PRIMARY KEY, 
            value blob 
            ) WITH caching = { 'keys' : 'ALL', 'rows_per_partition' : 'ALL' };";
  </code></pre>
  
  <p>session.execute(query);</p>
  
  <p><strong>QUERYING</strong></p>
  
  <pre><code>Select.Where statement1 = qb.select().from(TABLE_NAME).where(QueryBuilder.in(primaryKey, QueryBuilder.bindMarker()));
  pstatement= session.prepare(statement1).setConsistencyLevel(ConsistencyLevel.ONE);
  
  ByteBuffer test = (ByteBuffer) getCassandraKey(key, k);
  BoundStatement boundStatement = new BoundStatement(pstatement).bind(test);
  ResultSet results = session.execute(boundStatement);
  
  session.execute(statement);
  </code></pre>
  
  <p><strong>CLOSING</strong></p>
  
  <p>Session and Cluster are only closed once at the end of the application</p>
  
  <p><strong>System.log</strong></p>
  
  <p><em>NODE 1</em></p>
  
  <pre><code>INFO  [SharedPool-Worker-3] 2016-03-24 11:32:54,099 Message.java:605 - Unexpected exception during request; channel = [id: 0x63d065f4, /xx.xx.xx.xx:xxxxx :&gt; /xx.xx.xx.xx:xxxxx]
  java.io.IOException: Error while read(...): Connection reset by peer
          at io.netty.channel.epoll.Native.readAddress(Native Method) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.doReadBytes(EpollSocketChannel.java:675) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at io.netty.channel.epoll.EpollSocketChannel$EpollSocketUnsafe.epollInReady(EpollSocketChannel.java:714) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:326) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:264) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) ~[netty-all-4.0.23.Final.jar:4.0.23.Final]
          at java.lang.Thread.run(Thread.java:745) [na:1.8.0_65]
  INFO  [CompactionExecutor:8694] 2016-03-24 11:51:08,496 AutoSavingCache.java:386 - Saved KeyCache (2444860 items) in 2389 ms
  INFO  [IndexSummaryManager:1] 2016-03-24 11:51:24,546 IndexSummaryRedistribution.java:74 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 12:51:24,550 IndexSummaryRedistribution.java:74 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 13:51:24,553 IndexSummaryRedistribution.java:74 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 14:51:24,556 IndexSummaryRedistribution.java:74 - Redistributing index summaries
  INFO  [CompactionExecutor:8789] 2016-03-24 15:51:08,410 AutoSavingCache.java:386 - Saved KeyCache (2444860 items) in 2303 ms
  INFO  [IndexSummaryManager:1] 2016-03-24 15:51:24,559 IndexSummaryRedistribution.java:74 - Redistributing index summaries
  </code></pre>
  
  <p><em>NODE 2</em></p>
  
  <pre><code>INFO  [IndexSummaryManager:1] 2016-03-24 00:51:55,690 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 01:51:55,693 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 02:51:55,696 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [CompactionExecutor:8486] 2016-03-24 03:51:29,706 AutoSavingCache.java:386 - Saved KeyCache (2295232 items) in 2190 ms
  INFO  [IndexSummaryManager:1] 2016-03-24 03:51:55,698 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 04:51:55,701 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 05:51:55,704 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 06:51:55,707 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [CompactionExecutor:8583] 2016-03-24 07:51:29,623 AutoSavingCache.java:386 - Saved KeyCache (2295232 items) in 2099 ms
  INFO  [IndexSummaryManager:1] 2016-03-24 07:51:55,710 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 08:51:55,712 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 09:51:55,714 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 10:51:55,718 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [CompactionExecutor:8679] 2016-03-24 11:51:29,699 AutoSavingCache.java:386 - Saved KeyCache (2300663 items) in 2174 ms
  INFO  [IndexSummaryManager:1] 2016-03-24 11:51:55,721 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 12:51:55,724 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 13:51:55,727 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [IndexSummaryManager:1] 2016-03-24 14:51:55,730 IndexSummaryManager.java:257 - Redistributing index summaries
  INFO  [CompactionExecutor:8774] 2016-03-24 15:51:29,786 AutoSavingCache.java:386 - Saved KeyCache (2300663 items) in 2258 ms
  INFO  [IndexSummaryManager:1] 2016-03-24 15:51:55,733 IndexSummaryManager.java:257 - Redistributing index summaries
  </code></pre>