Well, a good example would be linear regression. We know pretty much all there is to know about its asymptotic behavior, and can prove that its results are &quot;maximum likelihood estimates.&quot;<p>Similarly, Bayesian statistics (a.k.a. probabilistic programming, graphical models, Bayes networks etc.) has a rock solid mathematical foundation – both the statistics behind it and the sampling algorithms (MCMC).<p>Of course, many mathematical statistical methods started off as hacks too. The invention of the L2 loss function, I imagine, must&#x27;ve gone a little like &quot;alright, let&#x27;s try to find the pattern here, let&#x27;s draw the line through this series of points that minimizes the sum of the distance of each point to the line... hm, I don&#x27;t like how this looks, let&#x27;s try squaring the distances instead and then minimize that sum so we punish large deviations from our line a bit more... oh, that looks better.&quot;