Why is the use of high order polynomials for regression discouraged? I've read many times on this site that high order polynomials (generally more than third) shouldn't be used in linear regression, unless there is a substantial justification to do so.
I understand the issues about extrapolation (and prediction at the boundaries).
Since extrapolation isn't important to me...

*

*Are high order polynomials also a bad way of approximating the underlying function within the range of the data points? (i.e. interpolation)

*If so, what problems are arising?

I don't mind being redirected to a good book or paper about this.
Thanks.

As an inveterate contrarian, I feel the need to amend the premise that high order polynomials shouldn't be used for interpolation. I would argue that the correct statement is "high order polynomials make poor interpolants unless properly regularized".
Indeed, it is quite popular (at least in academic circles) to conduct interpolation with polynomials of infinite degree, which is referred to as Kriging when a particular regularizer is used (the so called Reproducing Kernel Hilbert Space regularization, which intuitively speaking penalizes both a function's $L_2$ norm and its tendency to oscillate). And, of course, by infinite degree, we mean it is possible to construct a sequence of arbitrarily high degree polynomials which converge to this reasonable interpolant.
Additionally, recent work has shown that by increasing the degree to 2 or 3 times the size of the data (but keeping it finite), together with appropriate regularization, can still perform well. I can't find it just this moment, but this work was based on a similar discovery in the context of neural networks.
Generically, regularization prevents many of the issues brought up by other answerers.
