Question: In a world of magic/superpowers, would there be any demand for sports or games that didn't use magic? Magic here can be though of as a science and is mostly accessible to people who want to learn, but does have a bit of a steep learning curve at the beginning. It can be self taught if one wishes to put forth the effort and there's no ritual/incantation attached, so things are a bit more streamlined (Though, many will claim that you get better and more potent results with a good ol' fashion chant). In fact, magic here is so researched, it can be measured, enhanced and manipulated to the point where they have it down to a science.

Obviously, with the existence magic comes the potential for all new forms of sports and games. Flashy spectacles of daring and thrilling competition, teams raring to show of their adroit tactics and skills. People hold local tournaments and line the stadiums to see who will come out on top!

...That said, with such high octane thrillrides of magical sports and games, does anyone even care to watch or play the ones that don't involve spells and powers being flung around? Would things like football and baseball even have a chance against the likes of something such as that? 

Things that you can assume:

* It is measurable. This includes a person's potential. This way leagues can be created to divide according to aptitude. And it can all be regulated with well defined rules. (Usually people can go up in ability level after lots of practice, usually they don't go down unless they haven't used any magic in forever.)
* Not everyone knows how to use magic. Everyone can learn, but somepeople don't bother, because it doesn't pertain to their daily life and it would be a hassle. (Still most people do take the time, and a lot of times it can be taken in school as an elective)
* Magic has been around for a *very* long time. Thousands of years and is pretty ingrained in society for the most part.

So, do people care about the nonmagical variants of sports and games, or would they have died out when magic began to take over?

Answer: 
Whenever I read question like this, I can't help but to think that OP is too "anthropomorphic". People in your world can and probably would be different to us (unless introduction of magic is very recent) in terms of customs, morals and thinking in general. In this case, you can justify pretty much any difference between our cultures. 

In this specific case, you can easily have only non-magical sports and justify it by custom, honor, desire for purity and/or fairness or what have you. Alternatively, you have both types of sports, for reasons stated in other answers
The one spell you need cast to keep non-magical sports as a continuing source of pleasure to competitors and entertainment to spectators is a really good magic-suppressant spell over the playing field.