title: High CPU, possibly due to context switching?
body: <p>One of our servers is experiencing a very high CPU load with our application. We've looked at various stats and are having issues finding the source of the problem.</p>
  
  <p>One of the current theories is that there are too many threads involved and that we should try to reduce the number of concurrently executing threads. There's just one main thread pool, with 3000 threads, and a WorkManager working with it (this is Java EE - Glassfish). At any given moment, there are about 620 separate network IO operations that need to be conducted in parallel (use of java.NIO is not an option either). Moreover, there are roughly 100 operations that have no IO involved and are also executed in parallel.</p>
  
  <p>This structure is not efficient and we want to see if it is actually causing damage, or is simply bad practice. Reason being that any change is quite expensive in this system (in terms of man hours) so we need some proof of an issue.</p>
  
  <p>So now we're wondering if context switching of threads is the cause, given there are far more threads than the required concurrent operations. Looking at the logs, we see that on average there are 14 different threads executed in a given second. If we take into account the existence of two CPUs (see below), then it is 7 threads per CPU. This doesn't sound like too much, but we wanted to verify this.</p>
  
  <p>So - can we rule out context switching or too-many-threads as the problem?</p>
  
  <p>General Details:</p>
  
  <ol>
  <li>Java 1.5 (yes, it's old), running on CentOS 5, 64-bit, Linux kernel 2.6.18-128.el5</li>
  <li>There is only one single Java process on the machine, nothing else.</li>
  <li>Two CPUs, under VMware.</li>
  <li>8GB RAM</li>
  <li>We don't have the option of running a profiler on the machine.</li>
  <li>We don't have the option of upgrading the Java, nor the OS.</li>
  </ol>
  
  <p><strong>UPDATE</strong>
  As advised below, we've conducted captures of load average (using uptime) and CPU (using vmstat 1 120) on our test server with various loads. We've waited 15 minutes between each load change and its measurements to ensure that the system stabilized around the new load and that the load average numbers are updated:</p>
  
  <p>50% of the production server's workload: <a href="http://pastebin.com/GE2kGLkk" rel="noreferrer">http://pastebin.com/GE2kGLkk</a></p>
  
  <p>34% of the production server's workload: <a href="http://pastebin.com/V2PWq8CG" rel="noreferrer">http://pastebin.com/V2PWq8CG</a></p>
  
  <p>25% of the production server's workload: <a href="http://pastebin.com/0pxxK0Fu" rel="noreferrer">http://pastebin.com/0pxxK0Fu</a></p>
  
  <p>CPU usage appears to be reduced as the load reduces, but not on a very drastic level (change from 50% to 25% is not really a 50% reduction in CPU usage). Load average seems uncorrelated with the amount of workload.</p>
  
  <p>There's also a question: given our test server is also a VM, could its CPU measurements be impacted by other VMs running on the same host (making the above measurements useless)?</p>
  
  <p><strong>UPDATE 2</strong>
  Attaching the snapshot of the threads in three parts (pastebin limitations)</p>
  
  <p>Part 1: <a href="http://pastebin.com/DvNzkB5z" rel="noreferrer">http://pastebin.com/DvNzkB5z</a></p>
  
  <p>Part 2: <a href="http://pastebin.com/72sC00rc" rel="noreferrer">http://pastebin.com/72sC00rc</a></p>
  
  <p>Part 3: <a href="http://pastebin.com/YTG9hgF5" rel="noreferrer">http://pastebin.com/YTG9hgF5</a></p>