title: Random TimeoutException in Infinispan Hibernate L2 cache when put a load
body: <p>I am using Infinispan as a hibernate L2 cache. In a normal load system running without an issue. But when load put into the system the following exception was thrown from the cache cluster. Even though this not causing a functional impact reporting an ERROR seems a problem. </p>
  
  <p>Infinispan version : 5.3</p>
  
  <p>More info about the test : This hibernate L2 cache cluster had four nodes. During the test I didn't update the cache (Entities in the cache didn't update). Among four nodes two nodes heavily used the cache. But mostly I have seen this error in other two nodes. </p>
  
  <p><strong>Error log</strong>   </p>
  
  <pre><code>2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.interceptors.CallInterceptor - Executing command: RemoveCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@ac27701, value=null, flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], ignorePreviousValue=false}.
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.commands.write.RemoveCommand - Nothing to remove since the entry is null or we have a null entry
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.remoting.rpc.RpcManagerImpl - cosmos-db2-30512 invoking RemoveCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@ac27701, value=null, flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], ignorePreviousValue=false} to recipient list null with options RpcOptions{timeout=20000, unit=MILLISECONDS, fifoOrder=true, totalOrder=false, responseFilter=null, responseMode=SYNCHRONOUS, skipReplicationQueue=false}
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.remoting.transport.jgroups.JGroupsTransport - dests=null, command=SingleRpcCommand{cacheName='Order', command=RemoveCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@ac27701, value=null, flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], ignorePreviousValue=false}}, mode=SYNCHRONOUS, timeout=20000
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Replication task sending SingleRpcCommand{cacheName='Order', command=RemoveCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@ac27701, value=null, flags=[SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP], ignorePreviousValue=false}} to single recipient cosmos-db2-29372 with response mode GET_ALL
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.marshall.AdaptiveBufferSizePredictor - Next predicted buffer size for object type 'org.infinispan.commands.remote.SingleRpcCommand' will be 1536
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.marshall.VersionAwareMarshaller - Wrote version 510
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.VersionAwareMarshaller - Read version 510
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.marshall.AdaptiveBufferSizePredictor - Next predicted buffer size for object type 'org.infinispan.commands.remote.SingleRpcCommand' will be 512
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from factory
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.VersionAwareMarshaller - Read version 510
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.marshall.VersionAwareMarshaller - Wrote version 510
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher - Attempting to execute command: SingleRpcCommand{cacheName='Order', command=PutKeyValueCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2, value=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=2022, array=0x0301fe0409000000..}, cachedHashCode=1103892469}@40c6375, flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_ASYNCHRONOUS, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP, PUT_FOR_EXTERNAL_READ], putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true}} [sender=cosmos-db2-29372]
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop marshaller
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [remote-thread-0] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop marshaller
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.remoting.InboundInvocationHandlerImpl - Calling perform() on SingleRpcCommand{cacheName='Order', command=PutKeyValueCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2, value=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=2022, array=0x0301fe0409000000..}, cachedHashCode=1103892469}@40c6375, flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_ASYNCHRONOUS, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP, PUT_FOR_EXTERNAL_READ], putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true}}
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.commands.remote.BaseRpcInvokingCommand - Invoking command PutKeyValueCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2, value=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=2022, array=0x0301fe0409000000..}, cachedHashCode=1103892469}@40c6375, flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_ASYNCHRONOUS, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP, PUT_FOR_EXTERNAL_READ], putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true}, with originLocal flag set to false
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.interceptors.InvocationContextInterceptor - Invoked with command PutKeyValueCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2, value=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=2022, array=0x0301fe0409000000..}, cachedHashCode=1103892469}@40c6375, flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_ASYNCHRONOUS, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP, PUT_FOR_EXTERNAL_READ], putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true} and InvocationContext [org.infinispan.context.impl.NonTxInvocationContext@49559866]
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Marshallable type 'org.infinispan.marshall.MarshalledValue' known and is marshallable=true
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Marshallable type 'org.infinispan.marshall.MarshalledValue' known and is marshallable=true
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.statetransfer.StateTransferInterceptor - handleTopologyAffectedCommand for command PutKeyValueCommand{key=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2, value=MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=2022, array=0x0301fe0409000000..}, cachedHashCode=1103892469}@40c6375, flags=[ZERO_LOCK_ACQUISITION_TIMEOUT, FORCE_ASYNCHRONOUS, SKIP_CACHE_LOAD, SKIP_REMOTE_LOOKUP, PUT_FOR_EXTERNAL_READ], putIfAbsent=true, metadata=EmbeddedMetadata{version=null}, successful=true}
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.interceptors.locking.NonTransactionalLockingInterceptor - Are (cosmos-db2-30512) we the lock owners for key 'MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2'? true
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.util.concurrent.locks.LockManagerImpl - Attempting to lock MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2 with acquisition timeout of 0 millis
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.util.concurrent.locks.containers.ReentrantPerEntryLockContainer - Timed out attempting to acquire lock for key MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2 after 0 milliseconds
  2015-03-09 11:33:07 DEBUG [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.util.concurrent.locks.LockManagerImpl - Failed to acquire lock MarshalledValue{instance=&lt;serialized&gt;, serialized=ByteArray{size=1267, array=0x0301fe0409000000..}, cachedHashCode=1519}@545404b2, owner is Thread[remote-thread-0,5,main]
  2015-03-09 11:33:07 DEBUG [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.util.concurrent.locks.LockManagerImpl - This transaction (Thread[Incoming-1,cosmos-db2-30512,5,main]) already owned locks []
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.VersionAwareMarshaller - Read version 510
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Stop unmarshaller
  2015-03-09 11:33:07 ERROR [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.interceptors.InvocationContextInterceptor - ISPN000136: Execution error
  org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [0 milliseconds] on key [Order#1] for requestor [Thread[Incoming-1,cosmos-db2-30512,5,main]]! Lock held by [Thread[remote-thread-0,5,main]]
      at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:214)
      at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:197)
      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149)
      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:145)
      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:69)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:154)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
      at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
      at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.hibernate.cache.infinispan.impl.ClassLoaderAwareCache$ClassLoaderAwareCommandInterceptor.handleDefault(ClassLoaderAwareCache.java:79)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
      at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)
      at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)
      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:205)
      at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:111)
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:281)
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:233)
      at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
      at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
      at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
      at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:667)
      at org.jgroups.JChannel.up(JChannel.java:730)
      at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1019)
      at org.jgroups.stack.Protocol.up(Protocol.java:409)
      at org.jgroups.protocols.RSVP.up(RSVP.java:221)
      at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
      at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
      at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
      at org.jgroups.stack.Protocol.up(Protocol.java:409)
      at org.jgroups.protocols.SEQUENCER.up(SEQUENCER.java:270)
      at org.jgroups.stack.Protocol.up(Protocol.java:409)
      at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
      at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:792)
      at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:704)
      at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:382)
      at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
      at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
      at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
      at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
      at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
      at org.jgroups.protocols.Discovery.up(Discovery.java:379)
      at org.jgroups.protocols.TP.passMessageUp(TP.java:1405)
      at org.jgroups.protocols.TP$MyHandler.run(TP.java:1591)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  2015-03-09 11:33:07 ERROR [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.remoting.InboundInvocationHandlerImpl - Exception executing command
  org.infinispan.util.concurrent.TimeoutException: Unable to acquire lock after [0 milliseconds] on key [Order#1] for requestor [Thread[Incoming-1,cosmos-db2-30512,5,main]]! Lock held by [Thread[remote-thread-0,5,main]]
      at org.infinispan.util.concurrent.locks.LockManagerImpl.lock(LockManagerImpl.java:214)
      at org.infinispan.util.concurrent.locks.LockManagerImpl.acquireLockNoCheck(LockManagerImpl.java:197)
      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:149)
      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockKey(AbstractLockingInterceptor.java:145)
      at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitPutKeyValueCommand(AbstractLockingInterceptor.java:69)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.MarshalledValueInterceptor.visitPutKeyValueCommand(MarshalledValueInterceptor.java:154)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.statetransfer.StateTransferInterceptor.handleTopologyAffectedCommand(StateTransferInterceptor.java:216)
      at org.infinispan.statetransfer.StateTransferInterceptor.handleWriteCommand(StateTransferInterceptor.java:194)
      at org.infinispan.statetransfer.StateTransferInterceptor.visitPutKeyValueCommand(StateTransferInterceptor.java:136)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.CacheMgmtInterceptor.visitPutKeyValueCommand(CacheMgmtInterceptor.java:160)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.interceptors.IsMarshallableInterceptor.visitPutKeyValueCommand(IsMarshallableInterceptor.java:111)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:128)
      at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:92)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:120)
      at org.infinispan.interceptors.base.CommandInterceptor.handleDefault(CommandInterceptor.java:134)
      at org.hibernate.cache.infinispan.impl.ClassLoaderAwareCache$ClassLoaderAwareCommandInterceptor.handleDefault(ClassLoaderAwareCache.java:79)
      at org.infinispan.commands.AbstractVisitor.visitPutKeyValueCommand(AbstractVisitor.java:54)
      at org.infinispan.commands.write.PutKeyValueCommand.acceptVisitor(PutKeyValueCommand.java:83)
      at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:343)
      at org.infinispan.commands.remote.BaseRpcInvokingCommand.processVisitableCommand(BaseRpcInvokingCommand.java:61)
      at org.infinispan.commands.remote.SingleRpcCommand.perform(SingleRpcCommand.java:70)
      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleInternal(InboundInvocationHandlerImpl.java:122)
      at org.infinispan.remoting.InboundInvocationHandlerImpl.handleWithWaitForBlocks(InboundInvocationHandlerImpl.java:205)
      at org.infinispan.remoting.InboundInvocationHandlerImpl.handle(InboundInvocationHandlerImpl.java:111)
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.executeCommandFromLocalCluster(CommandAwareRpcDispatcher.java:281)
      at org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher.handle(CommandAwareRpcDispatcher.java:233)
      at org.jgroups.blocks.RequestCorrelator.handleRequest(RequestCorrelator.java:460)
      at org.jgroups.blocks.RequestCorrelator.receiveMessage(RequestCorrelator.java:377)
      at org.jgroups.blocks.RequestCorrelator.receive(RequestCorrelator.java:247)
      at org.jgroups.blocks.MessageDispatcher$ProtocolAdapter.up(MessageDispatcher.java:667)
      at org.jgroups.JChannel.up(JChannel.java:730)
      at org.jgroups.stack.ProtocolStack.up(ProtocolStack.java:1019)
      at org.jgroups.stack.Protocol.up(Protocol.java:409)
      at org.jgroups.protocols.RSVP.up(RSVP.java:221)
      at org.jgroups.protocols.FRAG2.up(FRAG2.java:182)
      at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
      at org.jgroups.protocols.FlowControl.up(FlowControl.java:434)
      at org.jgroups.stack.Protocol.up(Protocol.java:409)
      at org.jgroups.protocols.SEQUENCER.up(SEQUENCER.java:270)
      at org.jgroups.stack.Protocol.up(Protocol.java:409)
      at org.jgroups.protocols.pbcast.STABLE.up(STABLE.java:294)
      at org.jgroups.protocols.UNICAST3.removeAndDeliver(UNICAST3.java:792)
      at org.jgroups.protocols.UNICAST3.handleDataReceived(UNICAST3.java:704)
      at org.jgroups.protocols.UNICAST3.up(UNICAST3.java:382)
      at org.jgroups.protocols.pbcast.NAKACK2.up(NAKACK2.java:600)
      at org.jgroups.protocols.VERIFY_SUSPECT.up(VERIFY_SUSPECT.java:147)
      at org.jgroups.protocols.FD_ALL.up(FD_ALL.java:184)
      at org.jgroups.protocols.FD_SOCK.up(FD_SOCK.java:301)
      at org.jgroups.protocols.MERGE3.up(MERGE3.java:303)
      at org.jgroups.protocols.Discovery.up(Discovery.java:379)
      at org.jgroups.protocols.TP.passMessageUp(TP.java:1405)
      at org.jgroups.protocols.TP$MyHandler.run(TP.java:1591)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:745)
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [Incoming-1,cosmos-db2-30512] org.infinispan.remoting.InboundInvocationHandlerImpl - Unable to execute command, got invalid response ExceptionResponse
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [OOB-4,cosmos-db2-30512] org.infinispan.marshall.jboss.AbstractJBossMarshaller - Start unmarshaller after retrieving marshaller from thread local
  2015-03-09 11:33:07 TRACE [tid=cosmos-db2-150309112549721-1602836942-0-1] [OOB-4,cosmos-db2-30512] org.infinispan.marshall.VersionAwareMarshaller - Read version 510
  </code></pre>
  
  <p><strong>Hibernate L2 cache configuration</strong></p>
  
  <pre><code>&lt;?xml version="1.0" encoding="UTF-8"?&gt;
  &lt;infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="urn:infinispan:config:5.1"
              xsi:schemaLocation="urn:infinispan:config:5.1 http://www.infinispan.org/schemas/infinispan-config-5.1.xsd"&gt;
  
      &lt;global&gt;
          &lt;globalJmxStatistics enabled="true" cacheManagerName="H2L" jmxDomain="org.infinispan"
                               allowDuplicateDomains="true"/&gt;
          &lt;transport
                  transportClass="org.infinispan.remoting.transport.jgroups.JGroupsTransport"
                  clusterName="infinispan-hibernate-cluster"
                  distributedSyncTimeout="50000"
                  strictPeerToPeer="false"&gt;
              &lt;properties&gt;
                  &lt;property name="configurationFile" value="jgroups.xml"/&gt;
              &lt;/properties&gt;
          &lt;/transport&gt;
      &lt;/global&gt;
  
      &lt;default&gt;
      &lt;/default&gt;
  
      &lt;!-- entity/collection caching that uses replication --&gt;
      &lt;namedCache name="replicated-cache-entity"&gt;
          &lt;clustering mode="replication"&gt;
              &lt;stateTransfer fetchInMemoryState="false" timeout="20000"/&gt;
              &lt;sync replTimeout="20000"/&gt;
          &lt;/clustering&gt;
          &lt;!-- In distributed mode, Infinispan will store every entry on a subset of the nodes in the cache cluster
              (controlled by the parameter numOwners, which controls how many owners each entry will have).
              Theoretically reducing number of copies should increase the performance. Better to do a test with
              mode distribution as well. --&gt;
          &lt;!--&lt;clustering mode="distribution"&gt;--&gt;
              &lt;!--&lt;sync replTimeout="20000"/&gt;--&gt;
              &lt;!--&lt;hash numOwners="2" rehashEnabled="true"/&gt;--&gt;
          &lt;!--&lt;/clustering&gt;--&gt;
          &lt;locking isolationLevel="READ_COMMITTED" concurrencyLevel="1000"
                   lockAcquisitionTimeout="15000" useLockStriping="false"/&gt;
          &lt;!-- Eviction configuration. WakeupInterval defines how often the eviction thread runs, in milliseconds.
  0           means the eviction thread will never run. A separate executor is used for eviction in each cache. --&gt;
          &lt;!-- maxEntries should be fine tune depend on the entities used to caching. Right now caching the events there fore
               using a higher number --&gt;
          &lt;eviction maxEntries="1000" strategy="LRU"/&gt;
          &lt;expiration maxIdle="100000" wakeUpInterval="5000"/&gt;
          &lt;storeAsBinary enabled="true"/&gt;
          &lt;!--&lt;transaction useSynchronization="true"
          transactionMode="TRANSACTIONAL" autoCommit="false"
          lockingMode="OPTIMISTIC"/&gt;--&gt;
      &lt;/namedCache&gt;
  
    &lt;/infinispan&gt;
  </code></pre>
  
  <p>Appreciate your help.</p>