The answers above are fairly comprehensive, so I will just add a point by way of analogy which I think gives an intuition for why physicists (and scientists in general) prefer simpler theories and tend to be more inclined to believe they are somehow fundamentally correct if they give the right predictions.
It is common in science to have to perform "curve fitting", where one has many sets of measurements (to give a concrete example, let's say we have 50 measurements of the force between two newly discovered particles and the distance between them), generally with some noise and measurement error, and must determine how well they agree with some model. In our example, we might have a model of force where $F(r) = \frac{a}{r^2}$. Here we have a single parameter, $a$, so we have to answer two questions - which value of $a$ makes the model best fit our measured data, and once we've found it, how well does it fit? Usually it won't fit perfectly even if the model was correct because the measurements are never perfect, but generally the better the fit the more confident we can be in the model. We may compare two models by asking which can be made to better fit data.
However we must be careful with this line of reasoning, because of a phenomena called "overfitting". Let's say someone else comes along and claims that the force is described by a 100th degree polynomial, i.e. $F(r) = a_{100}r^{100} + a_{99}r^{99} + a_{98}r^{98} + ... + a_2r^2 + a_1r^1 + a_0r^0$. 484
As it turns out, given any set of $n$ pairs of measurements, we can perfectly fit that data with an $n$th degree polynomial, so that means our 100th degree polynomial can be perfectly fitted to our 50 measurements.
But obviously this doesn't mean the model is correct, in the case of our two new particles, because we could have always fit those data-points regardless of the underlying physics. This is known as "overfitting". This is also why most judgements of how well a model agrees with data don't just account for how closely it fits, but how many degrees of freedom the model had to fiddle with.
In general when discussing physical theories we may not have lots of easy to count numerical parameters. Whether one theory or the other has greater or fewer metaphorical parameters is non-obvious, and two theories where one seems more complicated may turn out to be mathematically, or even conceptually identical. Thus physicists are forced to rely on an intuitive sense of "elegance", a gut feeling that says a theory is "simple", "inevitable", that it is either right or wrong and can't be easily modified to fit observations that disagree slightly. I believe <PERSON> once summed up the feeling when he said something like "You can't put imperfections on a perfect thing, you have to come up with a new perfect thing instead". Perhaps one day we will figure out how to quantify this idea, but for now we're stuck with our guts. 484