What will the difference in speed be between an endurance geometry road bike and a race geometry road bike? I am looking to get a new road bike and am wondering if there will be a difference in speed between an endurance geometry bike like the Fuji Sportif or Specialized Diverge, and a race geometry bike like the Fuji Roubaix or Specialized Allez. I am looking to go as fast as possible but I also plan on going on 50+ mile rides so comfort is also a factor for me.  I also do not plan on doing many hardcore road races.
Other info: I am 140 lbs.
The main difference will be increased drag on the endurance model, as it is likely that you will have a higher handlebar position and this will increase your frontal area and in turn drag. However, if it is more comfortable in the more upright position, you may be faster as you are more efficient in putting down the power. 

If you are not racing regularly, my suggestion would be the endurance model, as you will be more comfortable, and if you want the low aggressive position that can be achieved by moving the stem/spacers around on the steerer tube.