Why do you need non-linear regression if you can use a linear one to fit any kind of curvature to your data? Polynomial regression fits a non-linear model to the data. But as a statistical estimation problem it's still linear in the sense that the regression function $h\left(\Theta, X\right)$ is linear in the unknown parameters $\Theta$.
When we use polynomial regression we actually give our linear model additional features like $X^2$ or $XY$. But with the same success you can give your model features like $\log\left(X\right)$ or $\exp\left(X\right)$, and after that apply least squares. So you can fit any kind of curvature to your data.
My question is: Why does non-linear regression assume a more general hypothesis space of functions - one that encompasses the hypothesis space of functions that you can get with linear regression? I mean why do we think that non-linear regression can fit more types of curvatures to the data than linear regression if linear regression itself (e.g. with polynomial or logarithmic features) can fit any curvature to the data?

*

*Model Parsimony

If you have a sine curve, you can approximate it to arbitrary accuracy with its series expansion.
I’d probably rather estimate the two parameters of $\mathbb E[y]= A\sin(Bx)$ than the many parameters in a long series expansion.
Note that, because the $B$ is inside the nonlinear sine function, you cannot create the estimated-frequency sine curve with a sine basis function; you would have to pick a $B$, rather than estimate it from the data.


*Interpretation

Parameters in the nonlinear equation can have interpretations of interest. In the above equation, $A$ is the amplitude and $B$ relates to the frequency. Perhaps you can wrestle with a long polynomial that approximates the sine curve in order to get at frequency and amplitude, but they are immediate from the nonlinear equation.
