What does interpolating the training set actually mean? I just read this article: Understanding Deep Learning (Still) Requires Rethinking Generalization
In section 6.1 I stumbled upon the following sentence

Specifically, in the overparameterized regime where the model capacity
greatly exceeds the training set size, fitting all the training
examples (i.e., interpolating the training set), including noisy ones,
is not necessarily at odds with generalization.

I do not fully understand the term "interpolating" in the context of fitting training data. Why do we speak of "interpolation" in this context? What does the term exactly mean here? Is there any other term that can be used instead?
In my understanding interpolation means the prediction within the training domain for some novel input that was not part of the training set.

In layman's terms, an interpolator will literally 'join the dots'.
Here's a simple graphical summary of what interpolation can do and why it can be awful. I'd like to stress that interpolation does play a useful role in statistics/ml but should be used carefully. The  black dots are training data and the red crossed are a similar dataset drawn form the same data generating process - they can be though of as a test set. 
We can see the in the left hand plot fits okay to both the training and test data. On the left I just used linear regression to fit a line to the data (just $2$ parameters). The curve in the right hand plot perfectly predicts the training set but looks nothing like the test set. I'd used an $11^{th}$ order polynomial (plus intercept) to fit the interpolator. Additionally, on the test set, the linear fit gives $MSE = 23.8$, the interpolator gives $MSE = 10350842349$ -- not good!
