title: How to fix "com.hazelcast.core.OperationTimeoutException: No response for 120000 ms" exceptions in hazelcast?
body: <p>I wrote a hazelcast that reads 100M datasets from a file and writes them into a MultiMap. I have 10 (locally running) members. After 70 mio nodes, I see this in my application log:</p>
  
  <pre><code>    ERROR 2014-02-19 03:03:56,168 [pool-1-thread-1] HazelcastMaster:190 - Could not finish calculations No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=Invoc
      ationImpl{ serviceName='hz:impl:multiMapService', op=com.hazelcast.multimap.operations.PutOperation@d8b737, partitionId=179, replicaIndex=0, tryCount=250, tryPauseMillis=500, invoke
      Count=1, callTimeout=60000, target=Address[144.76.62.99]:5704}, done=false}
      com.hazelcast.core.OperationTimeoutException: No response for 120000 ms. Aborting invocation! InvocationFuture{invocation=InvocationImpl{ serviceName='hz:impl:multiMapService', op=c
      om.hazelcast.multimap.operations.PutOperation@d8b737, partitionId=179, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[144.76.62.9
      9]:5704}, done=false}
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.waitForResponse(InvocationImpl.java:382)
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:294)
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:286)
              at com.hazelcast.multimap.MultiMapProxySupport.invoke(MultiMapProxySupport.java:276)
              at com.hazelcast.multimap.MultiMapProxySupport.putInternal(MultiMapProxySupport.java:52)
              at com.hazelcast.multimap.ObjectMultiMapProxy.put(ObjectMultiMapProxy.java:81)
              at de.komoot.spock.executor.HazelcastMultiMapStorage.add(HazelcastMultiMapStorage.java:31)
              at de.komoot.spock.impl.OsmNodesRepositoryImpl.add(OsmNodesRepositoryImpl.java:22)
              at de.komoot.spock.executor.HazelcastMaster$1.process(HazelcastMaster.java:96)
              at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.sendResultsToSink(PbfDecoder.java:106)
              at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.processBlobs(PbfDecoder.java:163)
              at org.openstreetmap.osmosis.pbf2.v0_6.impl.PbfDecoder.run(PbfDecoder.java:175)
              at de.komoot.spock.impl.PbfUrlReader.run(PbfUrlReader.java:63)
              at de.komoot.spock.executor.HazelcastMaster.call(HazelcastMaster.java:146)
              at java.util.concurrent.FutureTask.run(FutureTask.java:262)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:744)
  </code></pre>
  
  <p>And this had been send to STDERR:</p>
  
  <pre><code>    INFO: [144.76.62.99]:5710 [dev] memory.used=4.3G, memory.free=1.2G, memory.total=5.5G, memory.max=7.0G, memory.used/total=78.83%, memory.used/max=62.20%, load.process=0.00%, load.system=77.00%, load.systemAverage=1144.00%, thread.count=59, thread.peakCount=87, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.operation.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operation.size=0, executor.q.response.size=0, operations.remote.size=0, operations.running.size=0, proxy.count=2, clientEndpoint.count=0, connection.active.count=9, connection.count=9
      Feb 19, 2014 3:03:46 AM com.hazelcast.spi.Invocation
      WARNING: [144.76.62.99]:5701 [dev] No response for 120000 ms. InvocationFuture{invocation=InvocationImpl{ serviceName='hz:impl:multiMapService', op=com.hazelcast.multimap.operations.PutOperation@d8b737, partitionId=179, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[144.76.62.99]:5704}, done=false}
      Feb 19, 2014 3:03:47 AM com.hazelcast.spi.Invocation
      WARNING: [144.76.62.99]:5701 [dev] Asking if operation execution has been started: InvocationImpl{ serviceName='hz:impl:multiMapService', op=com.hazelcast.multimap.operations.PutOperation@d8b737, partitionId=179, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[144.76.62.99]:5704}
      Feb 19, 2014 3:03:52 AM com.hazelcast.spi.Invocation
      WARNING: [144.76.62.99]:5701 [dev] While asking 'is-executing': InvocationImpl{ serviceName='hz:impl:multiMapService', op=com.hazelcast.multimap.operations.PutOperation@d8b737, partitionId=179, replicaIndex=0, tryCount=250, tryPauseMillis=500, invokeCount=1, callTimeout=60000, target=Address[144.76.62.99]:5704}
      java.util.concurrent.TimeoutException
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.resolveResponse(InvocationImpl.java:432)
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:294)
              at com.hazelcast.spi.impl.InvocationImpl.isOperationExecuting(InvocationImpl.java:476)
              at com.hazelcast.spi.impl.InvocationImpl.access$1300(InvocationImpl.java:36)
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.waitForResponse(InvocationImpl.java:376)
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:294)
              at com.hazelcast.spi.impl.InvocationImpl$InvocationFuture.get(InvocationImpl.java:286)
              at com.hazelcast.multimap.MultiMapProxySupport.invoke(MultiMapProxySupport.java:276)
              at com.hazelcast.multimap.MultiMapProxySupport.putInternal(MultiMapProxySupport.java:52)
              at com.hazelcast.multimap.ObjectMultiMapProxy.put(ObjectMultiMapProxy.java:81)
  </code></pre>
  
  <p>(I redirected the output of all instances into one file).</p>
  
  <p>My question is now: How could I start finding the cause? It worked with a very simple dataset on my dev machine but now that I did a first bigger try I'm lost. What can I do to find the cause of the timeout?</p>
  
  <p>EDIT: I tried to give an example of what I do:</p>
  
  <pre><code>    MultiMap&lt;String, Node&gt;  data = instance.getMultiMap("MyMap");
  
      RunnableSource reader = new PbfUrlReader(input.openStream(), 4);
      Sink master = new Sink() {
  
              @Override
              public void process(EntityContainer entityContainer) {
                      Node node = ... //something made of entityContainer
                      String key = ... //key generated from entityContainer
                      data.put(key, node);
              }
      };
  
      reader.setSink(master);
      reader.run();
  </code></pre>