Resistance / immunity
This explanation has a number of interesting and negative side-effects that may not suit your world.
Every planet has a magical field. Life on any given planet has evolved to develop immunity to their own planet's field. It turns out that cellular growth while within a magical field results in too many mutations. Basically, magic acts something like radiation. In order for life to develop, therefore, evolution takes it down a path of very high resistance to the magical field of the planet that life is developing on. However, an evolutionary resistance to the native magic field also means you're not very good at using it.
So, an earth wizard using earth magic can exist, but very weakly. He is so immune to his own world's magic field that he can barely interact with it at all. 227
He can do simple cantrips but that's about it. However, this weak interaction was enough to allow wizards to stumble upon the ability to open rifts to other realms, and what a mother lode of easily manipulated magic they found there!
However, this explanation implies some potentially interesting side effects:
* Visiting another world is a hostile environment that will hurt you. Could be anything from "instant death by radiation exposure" to "you have a somewhat increased chance of developing cancer". Don't send your kids or pregnant wife to other realms. (The people may not understand exactly what's happening -- cellular damage -- but they recognize that going to another realm is "toxic" somehow.)
* Perhaps this can be mitigated by bringing your native magic field with you. Just as you open a rift to another realm to use their magic, you could travel foreign realms safely if you maintained a rift to your own world, surrounding yourself with your native field for safety. Maybe this is done by maintaining a spell or by carrying, say, a native lump of iron around with you.
* There's a potential "kryptonite" scenario here where assaulting a foreign wizard with his own planet's magic field hugely weakens him.
Have to think more about the implications but I like the idea that magic is a natural force, slightly different for each planet, which natives are necessarily, biologically immune to. 227