Question: Is it generally better to have a small number of fully upgraded rooms? Or have a larger number of not-so-upgraded ones? (Provided of course, that we have Power to spare.)

Wondering if the massive costs for upgrading are worth it. For example, I had a dweller training in a level 2 Strength Center with 2.00 hours to go. Then paid 4500 to upgrade it. The remaining time just went down by about 10 minutes! This seems pretty crazy since I could have built another 3-4 rooms with this cash.

Answer: 
I won't upgrade those rooms. You have later more than enough people to get others for hours in the training room. Buy 2-3 rooms and put the people in them instead one fast room. It's much cheaper and efficient.

By the way. You can further improve this. Just make a woman pregnant and put her inside the training room (not for the real world :D). This way you get more people while you train the existing ones. ;-)
Regardless of caps costs, you shouldn't upgrade training rooms. The reason is that higher level rooms get more severe incidents. The dwellers in your training rooms are usually low level, so if you don't upgrade your training rooms your rookies will be able to handle incidents unattended. If your training rooms are upgraded, you will have to heal your dwellers during the incident to keep them from dying. So even once you get more caps than you need, you should avoid upgrading your training rooms.