title: APIs slow down without any apparent bottlenecks on Cassandra nodes
body: <p>During Peak Load, Cassandra cluster performance deteriorates, though we dont see any spike in cpu utilization, Read or Write IO.<br />
  Read latencies go beyond 1 second</p>
  <p><a href="https://i.stack.imgur.com/pFJcv.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/pFJcv.png" alt="API Latency Graph" /></a></p>
  <p>Not able to scale the load when we increase the number of nodes in the cassandra cluster.
  When we increased the number of machines by 30% (reaching 36 nodes), we could not increase the load at least by the same percentage. It was much less than expected.
  Any hints on what else to debug would be very appreciated!</p>
  <p>Cluster Size: 27 nodes
  EC2 configurations:</p>
  <p>machine type:m4.2xlarge
  cpu: 8
  memory: 32 GB
  storage: EBS (1 TB)</p>
  <p>Cassandra Version: 3.11.4
  Replication factor : 3
  Total Data: around 500GB (without replication) ( It was 180GB when we had only Customer Data)</p>
  <p>Cassandra stats
  <a href="https://i.stack.imgur.com/ETWTH.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ETWTH.png" alt="enter image description here" /></a>
  <a href="https://i.stack.imgur.com/Q2OfV.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Q2OfV.jpg" alt="enter image description here" /></a></p>