Answers.Body: <p>The difference is simply that non-linear regression learns parameters that in some way control the non-linearity - e.g. any weight or bias that is applied <em>before</em> a non-linear function. </p>
  
  <p>For instance:</p>
  
  <p><span class="math-container">$$y = (w_1 x_1 + w_2 x_2)^2 + w_3$$</span> </p>
  
  <p>With such a function to learn, you <em>cannot</em> separate out transformed values of <span class="math-container">$w_1$</span> and <span class="math-container">$w_2$</span> and turn this into a linear function of just <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span>.   </p>
  
  <p>What you are describing as non-linearities in your examples are instead all applied by the machine learning engineer to <em>create</em> new candidate features for linear regression. This is not usually described as non-linear regression, but feature transformation or feature engineering.</p>
  
  <p>There is also a kind of middle ground where a central linear algorithm e.g. linear regression, is trained on many variations of the original features, by automated generation and filtering of transformed features. The most general variants of this approach are not hugely popular because they suffer from same risks of overfitting as non-linear models whilst not offering much in the way of improved performance. However, if you narrow down the types of feature and transformation combinations based on some knowledge of how you expect the target function to behave, it leads to many useful variants of linear regression - e.g. regression on fourier transforms, radial basis functions etc.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-22T17:15:43.933Z
Answers.LastActivityDate: 2019-12-22T17:29:23.817Z
Answers.LastEditDate: 2019-12-22T17:29:23.817Z
Answers.UserDisplayName: Neil Slater
Body: <p>In machine learning, I understand that linear regression assumes that parameters or weights in equation should be linear. For Example:</p>
  
  <p><span class="math-container">$$y = w_1x_1 + w_2x_2$$</span></p>
  
  <p>is a linear equation where <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span> are feature variables and <span class="math-container">$w_1$</span> and <span class="math-container">$w_2$</span> are parameters.</p>
  
  <p>Also</p>
  
  <p><span class="math-container">$$y = w_1(x_1)^2 + w_2(x_2)^2$$</span></p>
  
  <p>is also linear as parameters <span class="math-container">$w_1$</span> and <span class="math-container">$w_2$</span> are linear with respect to <span class="math-container">$y$</span>.</p>
  
  <p>Now, I read some articles stating that in the equation like</p>
  
  <p><span class="math-container">$$y = \log(w_1)x_1 + \log(w_2)x_2$$</span></p>
  
  <p>can also be made linear by considering other variables <span class="math-container">$v_1$</span> and <span class="math-container">$v_2$</span> as:</p>
  
  <p><span class="math-container">\begin{align}
  v_1 &amp;= \log(w_1)\\
  v_2 &amp;= \log(w_2)
  \end{align}</span></p>
  
  <p>Thus,</p>
  
  <p><span class="math-container">$$y = v_1x_1 + v_2x_2$$</span></p>
  
  <p>So, in this sense, any non-linear equation can be made linear, then what is non-linear regression here? I think I am missing something important here. I am a beginner in the field of Machine Learning. Can somebody help me?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-22T16:27:31.303Z
Comments.Text: non-linear regression is that dependent variable values does not depend on the linear combination. We can perform some transformations, like log or any transformation and make them linear. Though not every non linear relationship can be transformed into linear. Also even if we are able to do that.. that might be a complex transformation.
Comments.UserDisplayName: GadaaDhaariGeek
Tags: <machine-learning><comparison><linear-regression><non-linear-regression>
Title: What is the difference between linear and non-linear regression?
UserDisplayName: ka1shi
ThreadText: # What is the difference between linear and non-linear regression?
  
  (Asked by: username_0 on 2019-12-22)
  
  In machine learning, I understand that linear regression assumes that parameters or weights in equation should be linear. For Example:
  
  $$y = w\_1x\_1 + w\_2x\_2$$
  
  is a linear equation where $x\_1$ and $x\_2$ are feature variables and $w\_1$ and $w\_2$ are parameters.
  
  Also
  
  $$y = w\_1(x\_1)^2 + w\_2(x\_2)^2$$
  
  is also linear as parameters $w\_1$ and $w\_2$ are linear with respect to $y$.
  
  Now, I read some articles stating that in the equation like
  
  $$y = \log(w\_1)x\_1 + \log(w\_2)x\_2$$
  
  can also be made linear by considering other variables $v\_1$ and $v\_2$ as:
  
  \begin{align}
  v\_1 &= \log(w\_1)\\
  v\_2 &= \log(w\_2)
  \end{align}
  
  Thus,
  
  $$y = v\_1x\_1 + v\_2x\_2$$
  
  So, in this sense, any non-linear equation can be made linear, then what is non-linear regression here? I think I am missing something important here. I am a beginner in the field of Machine Learning. Can somebody help me?
  
  **Tags:** machine-learning, comparison, linear-regression, non-linear-regression
  > **Comments:**
  > **username_1**: non-linear regression is that dependent variable values does not depend on the linear combination. We can perform some transformations, like log or any transformation and make them linear. Though not every non linear relationship can be transformed into linear. Also even if we are able to do that.. that might be a complex transformation.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-12-22. Score: 2 | Accepted answer
  
  The difference is simply that non-linear regression learns parameters that in some way control the non-linearity - e.g. any weight or bias that is applied *before* a non-linear function.
  
  For instance:
  
  $$y = (w\_1 x\_1 + w\_2 x\_2)^2 + w\_3$$
  
  With such a function to learn, you *cannot* separate out transformed values of $w\_1$ and $w\_2$ and turn this into a linear function of just $x\_1$ and $x\_2$.
  
  What you are describing as non-linearities in your examples are instead all applied by the machine learning engineer to *create* new candidate features for linear regression. This is not usually described as non-linear regression, but feature transformation or feature engineering.
  
  There is also a kind of middle ground where a central linear algorithm e.g. linear regression, is trained on many variations of the original features, by automated generation and filtering of transformed features. The most general variants of this approach are not hugely popular because they suffer from same risks of overfitting as non-linear models whilst not offering much in the way of improved performance. However, if you narrow down the types of feature and transformation combinations based on some knowledge of how you expect the target function to behave, it leads to many useful variants of linear regression - e.g. regression on fourier transforms, radial basis functions etc.
  
  
  ---
  
  