Which upgrade should I consider for my usage: RAM or CPU? I plan to buy the new 2017 13" MacBook Pro with Touch Bar for the new semester. However, I'm trying to decide between upgrading the RAM or the CPU. 

I'm currently a Math major and plan to use it for things involving maths operations, like running Mathematica, factoring really big primes, graphing 3D contour plots etc. I'm also a bit of a gamer. Another consideration is that I really don't want the computer heating up, as I currently have the fan-less 2015 Retina MacBook and overheating is a common issue.

With this specific usage in mind, I'm really wondering what would benefit me the most: upgrading to the i7 processor or increasing the RAM to 16GB?
For starters, many users pay too much attention to overheating. Macs are designed to operate at a range of temperatures and to automatically protect themselves if they sense the temperature is above a safe level.

Secondly, you may get differences in opinion, but I'll try my best to answer this with facts.

While some i7 CPUs are quad-core, the ones offered on the 13" MBP are dual-core processors, just like the i5 CPUs.

Since all dual core i5 and i7 mobile processors support hyper-threading, there's actually very little difference between them in terms of a performance gap and battery usage. In fact, both options for this model also incorporate the same Intel Iris Plus Graphics GPU. 

My personal recommendation would be to upgrade both if you can find a way to afford it, as this will increase it's longevity in terms of future compatibility with software/hardware.

However, if you had to pick and choose what you upgraded, then the upgrade to 16GB RAM would be the most beneficial, as the performance boost on the i7 will benefit you less than the extra RAM would.