Question: I am early in this design but am imagining a fictional world in the age of exploration including classic fantasy tropes of monsters and magic, with renaissance era technology, that is cannons, muskets, clockwork and wacky renaissance war machines perhaps entwined with magic albeit to limited effect or making use of fantastical beasts. I would like the world to stay in this state of technology for as long as possible.

So I indeed ask, What would stop advancements in technology beyond this state? Specifically weaponry?

Answer: 
* **Magic sinks the scientific method.**  

At some point, [natural philosophers](https://en.wikipedia.org/wiki/Natural_philosophy#Natural_philosophy_in_the_early_modern_period) would have stopped *guessing* about [phlogiston](https://en.wikipedia.org/wiki/Phlogiston_theory) or [humors](https://en.wikipedia.org/wiki/Humorism) and started doing real science, using the [scientific method](https://en.wikipedia.org/wiki/Scientific_method). They think about things they see, they develop a coherent theory, they think of *experiments to falsify the theory* and *observations that would be predicted by the theory,* they observe and experiment, and they think and talk some more.  

Except that the world is just slightly magic. Observers influence the outcome of the experiment on a macro scale, not just on the quantum scale. The boiling point of water suddenly depends on how many people are watching, and if they want it to boil or not. At least a little. Black powder is more effective if it is mixed by people who *believe* in it.  

That sabotages the progress of science.
* **Dragons really like to nest on steam engines and blast furnaces.**  

They also enjoy village smithies and charcoal kilns, but not so much that a determined blacksmith cannot discourage them. But a steam engine, that's where monsters from the entire region will congregate. Better stay with water power and mostly wooden construction unless you have an army to protect your steam engine. Which means it doesn't pay to industrialize.
Religion or culture
===================

We can see both Religion and culture reject many advancements in the fields of science. Look at astronomy or biology with Religion for example. Though there were advancements, many advancements were rejected by the church in medieval Europe. The advancements were discredited and the people who made the discoveries put to death or incarcerated. It wasn’t that there wasn't progress, but much was made difficult.

Culture as well. Look at the Amish for example. But in a greater scale you can look at the whole of the USA. Advancements in trains or social norms in gun control are rejected because of their culture. They view that they have certain rights for their guns. They view their cars as essential as the indoctrination of the people did it's work, making any alternative unthinkable now that the whole infrastructure is setup to support only cars.

Human complacency with the current world is all you need. They obviously know the world is flat, they are the center of their world and nothing is going to change that. Anyone thinking differently or having proof can easily be rejected, preferably by hurning them.