How can we explain the "bad reputation" of higher-order polynomials? We all must have heard it by now - when we start learning about statistical models overfitting data, the first example we are often given is about "polynomial functions" (e.g., see the picture here):

We are warned that although higher-degree polynomials can fit training data quite well, they surely will overfit and generalize poorly to the test data.
Why does this happen? Is there a mathematical justification as to why (higher-degree) polynomial functions overfit the data? The closest explanation I could find online was something called "Runge's phenomenon", which suggests that higher-order polynomials tend to "oscillate" a lot - does this explain why polynomial functions are known to overfit data?
I understand that there is a whole field of "regularization" that tries to fix these overfitting problems (e.g., penalization can prevent a statistical model from "hugging" the data too closely) - but just using mathematical intuition, why are polynomials known to overfit the data?
In general, "functions" (e.g., the response variable you are trying to predict using machine learning algorithms) can be approximated using older methods like Fourier series, Taylor series and newer methods like neural networks. I believe that there are theorems that guarantee that Taylor series, polynomials and neural networks can "arbitrarily approximate" any function. Perhaps neural networks can promise smaller errors for simpler complexity?
But are there mathematical reasons behind higher-order polynomials (e.g., polynomial regression) being said to have a bad habit of overfitting, to the extent that they have become very unpopular? Is this solely explainable by Runge's phenomenon?
Reference:
Gelman, A. and Imbens, G. (2019) Why high order polynomials should not be used in regression discontinuity designs. Journal of Business and Economic Statistics 37(3), pp. 447-456. (An NBER working paper version is available here)

Using more terms means more degrees of freedom, hence more overfitting, but the real question is -- why are high order terms like $x^5$ so bad?
If you had enough data to estimate one parameter $a$, and no special knowledge about the domain, a practitioner would always start with $ax$ rather than $a x^5$ in their model exploration. Is it purely empirical, or is there some mathematical way to justify this choice?
When fitting discrete data you can show that low order terms are preferable to high order terms using worst case analysis.
https://arxiv.org/pdf/math/0410076.pdf
For random binary vectors of the form $\langle x_0,x_1,\ldots,x_n\rangle$, you can show that the best model relying on $k$ features of the form $\{x_i\}$ (first order) will result in a better expected fit when true generating distribution is picked adversarially, after your choice of features is revealed, than the best model relying on $k$ features of the form $\{x_i x_j\}$ (second order). I'm not aware of similar results for real valued $x$. The big obstacle is that for real-valued $x$ is that there is no longer agreement as to which measure to use over this space. Different choices of measure lead to different conclusions.
Perhaps there is some universality principle which suggests that low order terms are preferable for a typical case, which more relevant for real world, rather than "adversarial case" which is what is analysed in the paper.
