title: Profiling/optimising heavily multithreaded application
body: <p>I'm writing a performance-critical .NET application which makes heavy use of multithreading.</p>
  
  <p>Using the Visual Studio performance profiler, the top functions with Exclusive samples are:</p>
  
  <p><code>WaitHandle.WaitAny()</code> - 14.23%</p>
  
  <p><code>@JIT_MonReliableEnter@8</code> - 7.76%</p>
  
  <p><code>Monitor.Enter</code> - 5.09%</p>
  
  <p>Basically, my top 3 functions are working with threading primitives and out of my control to some extent I believe.  My work/processing routines are pretty small in comparison and I'm trying to increase performance.  I believe the algorithms involved are pretty sound, although I am reviewing them fairly frequently.</p>
  
  <p>My questions are:</p>
  
  <ul>
  <li>If there are 14.23% of CPU samples in these methods - is the CPU effectively 'idle' for most of those samples, i.e. just waiting on other threads?  Or is the idle part of the thread-waits not shown as a part of the profile trace [and the 27.08% shown in these 3 the sum of all overhead within those sync methods]?  (I can guess that this is mostly idle, but would appreciate some decent reference material behind answers to this one please)</li>
  <li>I have reviewed my locking schemes, however do these results indicate some particular bottleneck or technique I should look into for further optimisation?</li>
  <li>Is <code>WaitAny</code> quite poor in particular?  I use it heavily to check whether particular queue objects are readable/writable, but also checking an abort flag at the same time.  Is there a better way to do that?</li>
  </ul>