Answers.Body: <p>Regarding your first question, this is domain/task specific, and not always the case. My guess of why it happens in your situation (You did not specify your domain, so ill assume it's sometimes outside <span class="math-container">$(-1,1)$</span>) is that higher order polynomials increase much faster than lower order ones so it may fall under the trap where the coefficients have to be large to handle it. (e.g. if <span class="math-container">$x=2.0$</span> then <span class="math-container">$ 2^5*x^2 = x^7 \rightarrow 32*x^2 = x^7$</span>)  </p>
  
  <p>Now to answer why more points will cause less overfitting is sheerly because you have more equations. Your loss is generally some form of average distance accross points and predictions. Given <span class="math-container">$N$</span> points I can always find an <span class="math-container">$N-1$</span> order polynomial that fits it exactly. Instead of a formal proof I will provide a form in which you can easily generate this polynomial from <span class="math-container">$N$</span> <span class="math-container">$(x_i, y_i)$</span> pairs<br>
  <span class="math-container">$$ f(x) = \sum_{i=1}^{N}y_i\prod_{j=1\atop j \ != i}^N\frac{(x-x_j)}{(x_i-x_j)}$$</span>  </p>
  
  <p>As you can see, for each <span class="math-container">$x_i$</span> it will zero out all terms in the sum except one where the product become 1 making <span class="math-container">$f(x_i) = y_i$</span> by construction. So adding more points will prevent whatever optimization process you are learning to directly solve each point exactly, but instead try to find an algorithm that best generalizes to each of the points.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-02T14:31:20.190Z
Answers.LastActivityDate: 2019-08-02T14:31:20.190Z
Answers.UserDisplayName: mshlis
Body: <p>I'm reading the book Pattern Recognition and Machine Learning by Bishop, specifically the intro where he covers polynomial regression model. In short, let's say we generate <span class="math-container">$10$</span> data points using the function <span class="math-container">$\sin(2\pi x)$</span> and add some gaussian random noise to each observation. Now we pretend not knowing the generating function and try to fit a polynomial model to these points.</p>
  
  <p>As we increase the degree of the polynomial, it goes from underfitting (<span class="math-container">$d=1,2$</span>) to overfitting (<span class="math-container">$d=10$</span>). One thing the author notes is that the higher the degree of the polynomial, the higher the values of the coefficients (parameters). This is my first doubt: <strong>why does the size of the coefficients increase with the polynomial degree? And why is the size of the parameters related to overfitting?</strong></p>
  
  <p>Secondly, he states that even for degree <span class="math-container">$10$</span>, if we get sufficiently many data points (say <span class="math-container">$100$</span>), then the high degree polynomial will no longer overfit the data and should have comparatively better generalization performance. Second doubt: <strong>Why is this so?</strong></p>
  
Tags: <overfitting>
Title: Relation between size of parameters and complexity of model with overfitting
UserDisplayName: Shirish
ThreadText: # Relation between size of parameters and complexity of model with overfitting
  
  (Asked by: username_0 on 2019-08-02)
  
  I'm reading the book Pattern Recognition and Machine Learning by Bishop, specifically the intro where he covers polynomial regression model. In short, let's say we generate $10$ data points using the function $\sin(2\pi x)$ and add some gaussian random noise to each observation. Now we pretend not knowing the generating function and try to fit a polynomial model to these points.
  
  As we increase the degree of the polynomial, it goes from underfitting ($d=1,2$) to overfitting ($d=10$). One thing the author notes is that the higher the degree of the polynomial, the higher the values of the coefficients (parameters). This is my first doubt: **why does the size of the coefficients increase with the polynomial degree? And why is the size of the parameters related to overfitting?**
  
  Secondly, he states that even for degree $10$, if we get sufficiently many data points (say $100$), then the high degree polynomial will no longer overfit the data and should have comparatively better generalization performance. Second doubt: **Why is this so?**
  
  **Tags:** overfitting
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-02. Score: 1 | Accepted answer
  
  Regarding your first question, this is domain/task specific, and not always the case. My guess of why it happens in your situation (You did not specify your domain, so ill assume it's sometimes outside $(-1,1)$) is that higher order polynomials increase much faster than lower order ones so it may fall under the trap where the coefficients have to be large to handle it. (e.g. if $x=2.0$ then $ 2^5\*x^2 = x^7 \rightarrow 32\*x^2 = x^7$)
  
  Now to answer why more points will cause less overfitting is sheerly because you have more equations. Your loss is generally some form of average distance accross points and predictions. Given $N$ points I can always find an $N-1$ order polynomial that fits it exactly. Instead of a formal proof I will provide a form in which you can easily generate this polynomial from $N$ $(x\_i, y\_i)$ pairs  
  $$ f(x) = \sum\_{i=1}^{N}y\_i\prod\_{j=1\atop j \ != i}^N\frac{(x-x\_j)}{(x\_i-x\_j)}$$
  
  As you can see, for each $x\_i$ it will zero out all terms in the sum except one where the product become 1 making $f(x\_i) = y\_i$ by construction. So adding more points will prevent whatever optimization process you are learning to directly solve each point exactly, but instead try to find an algorithm that best generalizes to each of the points.
  
  
  ---
  
  