Answers.Body: <ol>
  <li><p>In some sense, you're right that a neural net is just another tool to fit data. However, it's <em>quite</em> the tool! There's this <a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow noreferrer">universal approximation theorem</a> saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all over if that's the right trend.</p>
  </li>
  <li><p>The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has badly overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a few matrix products (the <span class="math-container">$\hat{\beta}=(X^TX)^{-1}X^Ty$</span>).</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T16:14:18.887Z
Answers.Text: Not an expert but is the universal approximation theorem really saying something very special about neural networks? Paraphrasing a neural network expert: "The universal approximation theorem does NOT guarantee that you will converge to an arbitrarily good approximation of a wide class functions. It just means that this approximation is in the search space. There is no guarantee you will converge to it and in fact you almost never will." This sounds very similar to guarantees that are available when fitting with many other bases (sin/cos, polynomials, etc...)
Answers.UserDisplayName: Kvothe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T16:36:40.403Z
Answers.Text: @Kvothe It is not clear if that quote refers to convergence in terms of training (converging to the optimum for a given architecture) or convergence of network to a function as more and more nodes are added to it. I am on board with the former but not the latter. The universal approximation theorems say, much like the Stone-Weierstrass theorem for polynomials or Carlson's theorem for Fourier series, that you can approximate any function (of a particular class, specified by the theorem details) by making a sufficiently complex neural network.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T16:42:05.613Z
Answers.Text: I think I meant the former. In the latter part of your reply: Are you agreeing that there is nothing too special about neural networks and that universal approximation theorem basically just gives you a completeness of your basis that you also get in most other fitting methods?
Answers.UserDisplayName: Kvothe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-14T16:58:00.670Z
Answers.Text: @Kvothe Completeness as in Cauchy completeness?
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T02:01:23.007Z
Answers.Text: I might be incorrectly borrowing language from intuitively similar spaces (e.g. Hilbert spaces). With "complete basis" in this case I mean a set of functions which give an arbitrarily good approximation for any function in a certain class of functions (the space for which these functions act as a "basis").
Answers.UserDisplayName: Kvothe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T02:04:26.020Z
Answers.Text: @Kvothe The various universal approximation theorems give the conditions under which a neural network can approximate a function to any degree of accuracy. That sounds like what you’re saying, yes.
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T02:07:26.813Z
Answers.Text: Right, my point being that there are many alternatives, making neural nets not that special. It is not that clear from your answer what advantages it has over other approaches that come with very similar theorems. For that matter it sounds like you would also say that polynomial fits or Fourier series are also "quite the tool!". What really sets neural networks apart?
Answers.UserDisplayName: Kvothe
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-17T02:14:16.903Z
Answers.Text: @Kvothe To a large extent, I don’t think neural networks are that special, just popular (warranted or not). Perhaps you could consider posting your question formally, either here or on the statistics Stack, [Cross Validated](https://stats.stackexchange.com/), where a more thorough answer could be posted (not just by me).
Answers.UserDisplayName: Dave
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-15T21:10:15.903Z
Answers.LastActivityDate: 2023-07-14T16:33:40.880Z
Answers.LastEditDate: 2023-07-14T16:33:40.880Z
Answers.UserDisplayName: Dave
Body: <p>For simplicity, let's assume we want to solve a regression problem, where we have one independent variable and one dependent variable, which we want to predict. Let's also assume that there is a nonlinear relationship between the independent and dependent variables.</p>
  <p>No matter the way we do it, we just need to build a proper curved line based on existing observations, such that the prediction is the best.</p>
  <p>I know we can solve this problem with neural networks, but I also know other ways to create such curves. For example:</p>
  <ol>
  <li><p>splines</p>
  </li>
  <li><p>kriging</p>
  </li>
  <li><p>lowess</p>
  </li>
  <li><p>Something I think would also work (do not know if exists): fitting curve using a series of Fourier sine waves, and so on</p>
  </li>
  </ol>
  <p>My questions are:</p>
  <ol>
  <li><p>Is it true that neural networks are just one of the ways to fit a non-linear curve to the data?</p>
  </li>
  <li><p>What are the advantages and disadvantages of choosing a neural network over other approaches? (maybe it becomes better when I have many independent variables, and another little guess: maybe the neural network is better in omitting the effect of linear dependent input variables?)</p>
  </li>
  </ol>
  
Tags: <neural-networks><machine-learning><comparison><non-linear-regression><curve-fitting>
Title: What is the difference between neural networks and other ways of curve fitting?
UserDisplayName: Igor
ThreadText: # What is the difference between neural networks and other ways of curve fitting?
  
  (Asked by: username_0 on 2020-10-15)
  
  For simplicity, let's assume we want to solve a regression problem, where we have one independent variable and one dependent variable, which we want to predict. Let's also assume that there is a nonlinear relationship between the independent and dependent variables.
  
  No matter the way we do it, we just need to build a proper curved line based on existing observations, such that the prediction is the best.
  
  I know we can solve this problem with neural networks, but I also know other ways to create such curves. For example:
  
  1. splines
  2. kriging
  3. lowess
  4. Something I think would also work (do not know if exists): fitting curve using a series of Fourier sine waves, and so on
  
  My questions are:
  
  1. Is it true that neural networks are just one of the ways to fit a non-linear curve to the data?
  2. What are the advantages and disadvantages of choosing a neural network over other approaches? (maybe it becomes better when I have many independent variables, and another little guess: maybe the neural network is better in omitting the effect of linear dependent input variables?)
  
  **Tags:** neural-networks, machine-learning, comparison, non-linear-regression, curve-fitting
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-15. Score: 4 | Accepted answer
  
  1. In some sense, you're right that a neural net is just another tool to fit data. However, it's *quite* the tool! There's this [universal approximation theorem](https://en.wikipedia.org/wiki/Universal_approximation_theorem) saying that, under decent conditions, a neural network can get as close as you want to a wide class of functions. This means that you can get the network to give you complicated shapes with squiggles all over if that's the right trend.
  2. The universal approximation theorem is a big upside. You don't have to specify that you want to model with sine curves or a particular type of spline. You just let the computer figure that out for you. The result is the ability to model complex patterns and make accurate predictions. The drawback is that the modeling can pick up on coincidences in the data that look like a trend but are not. This causes overfitting. When your goal is to make accurate predictions, a model that has badly overfit does nothing for you. A second drawback is that neural networks are hard to interpret. A third drawback is that they can take a long time to train, while a linear regression is just a matrix inversion and a few matrix products (the $\hat{\beta}=(X^TX)^{-1}X^Ty$).
  
  > **Comments:**
  > **username_2**: Not an expert but is the universal approximation theorem really saying something very special about neural networks? Paraphrasing a neural network expert: "The universal approximation theorem does NOT guarantee that you will converge to an arbitrarily good approximation of a wide class functions. It just means that this approximation is in the search space. There is no guarantee you will converge to it and in fact you almost never will." This sounds very similar to guarantees that are available when fitting with many other bases (sin/cos, polynomials, etc...)  
  > **username_1**: @username_2 It is not clear if that quote refers to convergence in terms of training (converging to the optimum for a given architecture) or convergence of network to a function as more and more nodes are added to it. I am on board with the former but not the latter. The universal approximation theorems say, much like the Stone-Weierstrass theorem for polynomials or Carlson's theorem for Fourier series, that you can approximate any function (of a particular class, specified by the theorem details) by making a sufficiently complex neural network.  
  > **username_2**: I think I meant the former. In the latter part of your reply: Are you agreeing that there is nothing too special about neural networks and that universal approximation theorem basically just gives you a completeness of your basis that you also get in most other fitting methods?  
  > **username_1**: @username_2 Completeness as in Cauchy completeness?  
  > **username_2**: I might be incorrectly borrowing language from intuitively similar spaces (e.g. Hilbert spaces). With "complete basis" in this case I mean a set of functions which give an arbitrarily good approximation for any function in a certain class of functions (the space for which these functions act as a "basis").  
  > **username_1**: @username_2 The various universal approximation theorems give the conditions under which a neural network can approximate a function to any degree of accuracy. That sounds like what you’re saying, yes.  
  > **username_2**: Right, my point being that there are many alternatives, making neural nets not that special. It is not that clear from your answer what advantages it has over other approaches that come with very similar theorems. For that matter it sounds like you would also say that polynomial fits or Fourier series are also "quite the tool!". What really sets neural networks apart?  
  > **username_1**: @username_2 To a large extent, I don’t think neural networks are that special, just popular (warranted or not). Perhaps you could consider posting your question formally, either here or on the statistics Stack, [Cross Validated](https://stats.stackexchange.com/), where a more thorough answer could be posted (not just by me).  
  
  ---
  
  