Question: In a world of magic/superpowers, would there be any demand for sports or games that didn't use magic? Magic here can be though of as a science and is mostly accessible to people who want to learn, but does have a bit of a steep learning curve at the beginning. It can be self taught if one wishes to put forth the effort and there's no ritual/incantation attached, so things are a bit more streamlined (Though, many will claim that you get better and more potent results with a good ol' fashion chant). In fact, magic here is so researched, it can be measured, enhanced and manipulated to the point where they have it down to a science.

Obviously, with the existence magic comes the potential for all new forms of sports and games. Flashy spectacles of daring and thrilling competition, teams raring to show of their adroit tactics and skills. People hold local tournaments and line the stadiums to see who will come out on top!

...That said, with such high octane thrillrides of magical sports and games, does anyone even care to watch or play the ones that don't involve spells and powers being flung around? Would things like football and baseball even have a chance against the likes of something such as that? 

Things that you can assume:

* It is measurable. This includes a person's potential. This way leagues can be created to divide according to aptitude. And it can all be regulated with well defined rules. (Usually people can go up in ability level after lots of practice, usually they don't go down unless they haven't used any magic in forever.)
* Not everyone knows how to use magic. Everyone can learn, but somepeople don't bother, because it doesn't pertain to their daily life and it would be a hassle. (Still most people do take the time, and a lot of times it can be taken in school as an elective)
* Magic has been around for a *very* long time. Thousands of years and is pretty ingrained in society for the most part.

So, do people care about the nonmagical variants of sports and games, or would they have died out when magic began to take over?

Answer: 
It is practically guaranteed that you would still have non-magical sports and games.

The fairly simple reason is that with most sports, the rules are set up so it won't be too easy.  

When playing darts, the distance to the board is there to make it harder to hit the mark.  

In soccer you may not use your hands. In baseball, you cannot use a computer controlled gun turret to shoot the ball, although that would be easier than using a length of wood. In bicycle races, you cannot use motorcycles, although they would be faster.

Of course we also have motorcycle races, so in your world you will also have (enter magic-powered sport of your liking), but it will go alongside non-magical sports, it won't replace them.

Some people will prefer some of the non-magical sports, others will prefer other stuff, and, as a side note, of course they will all point at the others and say "that's not really sports".
"Any sufficiently advanced technology is indistinguishable from magic" - Arthur C. Clarke

We have a world of wonder and excitement right now. Roller coasters, video games, skydiving. There are places to find thrills that we never imagined years ago. E-sports are a thing now. We have a computers that are the best in the world at ancient board games.

Even with all this wonder, we still love soccer and hockey and football. We will always have those: they are a part of our culture, and we're not giving them up. There will be magical sports, but there is another joy in restricting your ability with strict rules for the challenge of it.