What is overfitting while building model? What exactly is overfitting while building models ?

I like this simple mathematical explanation for the regression case so I'll leave it here:
Overfitting is a consequence of an imbalance between the following three factors:

*

*Number of training samples N. If you have many training samples, you should learn well, even if the model is complex. Conversely, if the model is complex but does not have enough training samples, you will overfit it.

*Model order d. This refers to the complexity of the model (d <= N). If your training set is too small, you need to use a less complex model.

*Noise variance σ2. This refers to the variance of the error added to the data. If σ increases, it is inevitable that the fitting will become more difficult. Hence it would require more training samples, and perhaps a less complex model would work better.

In short:
MSEtrain = σ2 (1 - d/N)
MSEtest = σ2 (1 + d/N)
