WindowServer high CPU on Yosemite This is a follow up to So my Yosemite install is slow - what to do?

Since installing Yosemite a week or so ago my machine is very sluggish. I am a web developer so I change windows/apps a lot. Changing apps makes my machine hang for 2 or more seconds.

After observing the Activity Monitor I can see that WindowServer is taking up a lot of CPU when I move windows and also when I am just using one app.  I have tried closing apps to determine which, if any, is causing the issue but haven't come up with anything useful yet.

Here is an example.



Even right now as I type is flips between 10% and 20% usage. I am only typing!

How can I fix this?
I had a similar issue with high cpu usage in WindowServer which I managed to get back to something more normal by removing any items in my menu bar that were making high frequency drawing updates. 

In my case it was the Network Monitor from Little Snitch that seemed to be the biggest culprit.