Question: Is it generally better to have a small number of fully upgraded rooms? Or have a larger number of not-so-upgraded ones? (Provided of course, that we have Power to spare.)

Wondering if the massive costs for upgrading are worth it. For example, I had a dweller training in a level 2 Strength Center with 2.00 hours to go. Then paid 4500 to upgrade it. The remaining time just went down by about 10 minutes! This seems pretty crazy since I could have built another 3-4 rooms with this cash.

Answer: 
I won't upgrade those rooms. You have later more than enough people to get others for hours in the training room. Buy 2-3 rooms and put the people in them instead one fast room. It's much cheaper and efficient.

By the way. You can further improve this. Just make a woman pregnant and put her inside the training room (not for the real world :D). This way you get more people while you train the existing ones. ;-)
Build across then in. It's much cheaper to build across 2-3 rooms than it is to upgrade. You DO want to upgrade though, it increases your storage which is your main buffer against disaster. Also by building across first you save yourself the grief of having to delete a room just to expand one you really need.