Question: I am early in this design but am imagining a fictional world in the age of exploration including classic fantasy tropes of monsters and magic, with renaissance era technology, that is cannons, muskets, clockwork and wacky renaissance war machines perhaps entwined with magic albeit to limited effect or making use of fantastical beasts. I would like the world to stay in this state of technology for as long as possible.

So I indeed ask, What would stop advancements in technology beyond this state? Specifically weaponry?

Answer: 
**No oil, no progress.**

The modern world is only possible because we have vast reserves of oil and coal underground - courtesy of pines and shrimp which fossilized in prior epochs. Without coal, even if electricity was discovered there would have been no resource energy rich enough to run large scale power plants. Without oil, we could never have progressed further then smoggy powerplants and into the era of personal, horseless, vehicles and trade.

Simply state that there are no oil reserves on your world and limited coal mines and your civilization is then stuck in the renaissance.

On the side of weaponry all I can add is that it wasn't the military that came up with early improvements to guns. It was enthusiast duck hunters in between the wars. The militaries were content with powder loaded muskets and trying to built continually larger cannons for a long time. The idea of getting off more then a single precision shot from a gun was engineered by sports hunters as a way to hit more ducks. So... Remove ducks from your world?
Magic has to be superior to technology.
---------------------------------------

The development of weaponry didn't happen in a vacuum, it was in response to battlefield conditions. People stuck with firearms and continued to improve them because they were effective against the defenses they encountered. So if you don't want certain types of weapons to advance, make them ineffective. The obvious solution is to have magic render bullets as harmless with trivial protective wardings, or something along those lines. Something where the magical protection is far easier than the mundane weapon it defeats.

This creates a new problem: why are people using ineffective weapons at all? At what point did magical protections become common in warfare? That's the point where you'd see mundane weapon development stall out. So think about when this happens, and why. What changed to make magical protection commonplace? And how do armies defeat it? Probably with magical weapons of some kind.

So your task is to think about how this society uses magic to wage war, and to devise reasons for why building more advanced machines never seemed like a feasible or worthwhile idea to get the edge over the enemy. I think for this to be the case **magical solutions have to be cheaper, easier to use, and more readily available than the means to mass produce machines**, as well as more effective.

This should be your guiding principle when thinking about how magic is used if you want it to essentially replace the development of modern technology. It should be accessible to people at all levels of society and be fundamentally superior to the technology it replaces. If technology is easier for a majority of people to access and can defeat magic, nobody would use magic, especially in war.