Question: Is it generally better to have a small number of fully upgraded rooms? Or have a larger number of not-so-upgraded ones? (Provided of course, that we have Power to spare.)

Wondering if the massive costs for upgrading are worth it. For example, I had a dweller training in a level 2 Strength Center with 2.00 hours to go. Then paid 4500 to upgrade it. The remaining time just went down by about 10 minutes! This seems pretty crazy since I could have built another 3-4 rooms with this cash.

Answer: 
I won't upgrade those rooms. You have later more than enough people to get others for hours in the training room. Buy 2-3 rooms and put the people in them instead one fast room. It's much cheaper and efficient.

By the way. You can further improve this. Just make a woman pregnant and put her inside the training room (not for the real world :D). This way you get more people while you train the existing ones. ;-)
Later on, with large vaults, I'd say time is more important than money, and every little bit helps and are well worth upgrading the training rooms. Especially once you realize that having a Dweller's Endurance maxed out BEFORE they start leveling up drastically increases their hit points and start trying to max out level 1 dwellers before sending them to any tasks. At that point, the wait can get painful. 

That being said, from what I've been able to find, it seems like the # of dwellers in a room training at the time may have a larger effect than the level of the room. 

I haven't been able to find any greatly detailed or thoroughly researched numbers, but based on [this reddit post and accompanying spreadsheet](https://www.reddit.com/r/foshelter/comments/3auqxd/fallout_shelter_approximate_training_times_for/) going from stat 9 to 10 in a tier1 room vs tier3 room is only a savings of 1 hour 18 minutes. Whereas having a tier 2 room with 6 dwellers in it vs just 1 dweller going from 9 to 10 is a savings of just shy of 2 hours. 

So, it may be more beneficial to build 3 block training rooms and try to have 6 dwellers training at once, vs paying for all the upgrades. Of course, having 3 wide, fully upgraded training rooms with 6 dwellers training is going to give you the shortest training times possible.