Why does a decimal sized 26" tire fit my rims but a fractional sized 26" tire doesn't? I'm replacing some 26 x 2.10" mountain bike tires that I have on my bike with some thinner 26 x 1 3/8" road bike tires for commuting, however somehow even though they both say that they are 26" the new road bike tires are too big for the rim!

I can take the new tires back, what should I look for when picking out a new set of tires?
You ask what to look for when buying MTB rims.

The most obvious thing would be ask for Mountain Bike tires (slick tires in your case) at the shop. Just using the words "mountain bike" is enough for any shop attendant.

Now if you want to take a tire and, just by looking, tell if it fits your rims, you must look for the ETRTO code, which states the inner diameter of the tire bead in milimeters, which is 559 for MTB.

So you have 47-559 (read right now from one of my tires) to be equivalent of 26x47 (47mm wide, or roughly 1.9)

Hope this helps