Well X11 in general was memory bandwidth hungry, but you drop down to 720p (closer to the 1154 x 900 size) and you get a bit better. Also if you could get the Pi to run with a black and white frame buffer it might help.