Question: I am early in this design but am imagining a fictional world in the age of exploration including classic fantasy tropes of monsters and magic, with renaissance era technology, that is cannons, muskets, clockwork and wacky renaissance war machines perhaps entwined with magic albeit to limited effect or making use of fantastical beasts. I would like the world to stay in this state of technology for as long as possible.

So I indeed ask, What would stop advancements in technology beyond this state? Specifically weaponry?

Answer: 
* **Magic sinks the scientific method.**  

At some point, [natural philosophers](https://en.wikipedia.org/wiki/Natural_philosophy#Natural_philosophy_in_the_early_modern_period) would have stopped *guessing* about [phlogiston](https://en.wikipedia.org/wiki/Phlogiston_theory) or [humors](https://en.wikipedia.org/wiki/Humorism) and started doing real science, using the [scientific method](https://en.wikipedia.org/wiki/Scientific_method). They think about things they see, they develop a coherent theory, they think of *experiments to falsify the theory* and *observations that would be predicted by the theory,* they observe and experiment, and they think and talk some more.  

Except that the world is just slightly magic. Observers influence the outcome of the experiment on a macro scale, not just on the quantum scale. The boiling point of water suddenly depends on how many people are watching, and if they want it to boil or not. At least a little. Black powder is more effective if it is mixed by people who *believe* in it.  

That sabotages the progress of science.
* **Dragons really like to nest on steam engines and blast furnaces.**  

They also enjoy village smithies and charcoal kilns, but not so much that a determined blacksmith cannot discourage them. But a steam engine, that's where monsters from the entire region will congregate. Better stay with water power and mostly wooden construction unless you have an army to protect your steam engine. Which means it doesn't pay to industrialize.
**No oil, no progress.**

The modern world is only possible because we have vast reserves of oil and coal underground - courtesy of pines and shrimp which fossilized in prior epochs. Without coal, even if electricity was discovered there would have been no resource energy rich enough to run large scale power plants. Without oil, we could never have progressed further then smoggy powerplants and into the era of personal, horseless, vehicles and trade.

Simply state that there are no oil reserves on your world and limited coal mines and your civilization is then stuck in the renaissance.

On the side of weaponry all I can add is that it wasn't the military that came up with early improvements to guns. It was enthusiast duck hunters in between the wars. The militaries were content with powder loaded muskets and trying to built continually larger cannons for a long time. The idea of getting off more then a single precision shot from a gun was engineered by sports hunters as a way to hit more ducks. So... Remove ducks from your world?