Question: Is it generally better to have a small number of fully upgraded rooms? Or have a larger number of not-so-upgraded ones? (Provided of course, that we have Power to spare.)

Wondering if the massive costs for upgrading are worth it. For example, I had a dweller training in a level 2 Strength Center with 2.00 hours to go. Then paid 4500 to upgrade it. The remaining time just went down by about 10 minutes! This seems pretty crazy since I could have built another 3-4 rooms with this cash.

Answer: 
Regardless of caps costs, you shouldn't upgrade training rooms. The reason is that higher level rooms get more severe incidents. The dwellers in your training rooms are usually low level, so if you don't upgrade your training rooms your rookies will be able to handle incidents unattended. If your training rooms are upgraded, you will have to heal your dwellers during the incident to keep them from dying. So even once you get more caps than you need, you should avoid upgrading your training rooms.
Build across then in. It's much cheaper to build across 2-3 rooms than it is to upgrade. You DO want to upgrade though, it increases your storage which is your main buffer against disaster. Also by building across first you save yourself the grief of having to delete a room just to expand one you really need.