Answers.Body: <p>You can indeed fit a polynomial to your labelled data, which is known as <a href="https://en.wikipedia.org/wiki/Polynomial_regression" rel="nofollow noreferrer">polynomial regression</a> (which can e.g. be done with the function <a href="https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html" rel="nofollow noreferrer"><code>numpy.polyfit</code></a>). One apparent limitation of polynomial regression is that, in practice, you need to assume that your data follows some specific polynomial of some degree <span class="math-container">$n$</span>, i.e. you assume that your data has the form of the polynomial that you choose, which may not be true.</p>
  <p>When you use a neural network to solve a classification or regression problem, you also need to choose the activation functions, the number of neurons, how they are connected, etc., so you also need to limit the number and type of functions that you can learn with neural networks, i.e. the <em>hypothesis space</em>.</p>
  <p>Now, it is not necessarily a bad thing to limit the hypothesis space. In fact, <em>learning</em> is generally an <em>ill-posed problem</em>, i.e. in simple terms, there could be multiple solutions or no solutions at all (<a href="https://www.mit.edu/%7E9.520/spring12/slides/class02/class02.pdf" rel="nofollow noreferrer">and other problems</a>), so, actually, you often need to limit the hypothesis space to find some useful solutions (e.g. solutions that generalise better to unseen data). <em>Regularisations techniques</em> are ways of constraining the learning problem, and the hypothesis space (i.e. the set of functions that your learning algorithm can choose from), and thus making the learning problem well-posed.</p>
  <p>Neural networks are not preferred over polynomial regression because they are theoretically more powerful. In fact, both can approximate any continuous function [<a href="https://link.springer.com/content/pdf/10.1007/BF02551274.pdf" rel="nofollow noreferrer">1</a>], but these are just theoretical results, i.e. these results do not give you the magical formula to choose the most appropriate neural network or polynomial that best approximates the desired unknown function.</p>
  <p>In practice, neural networks have been proven to <strong>effectively solve many tasks</strong> (e.g. translation of natural language, playing go or atari games, image classification, etc.), so I would say that this is the main reason they are widely studied and there is a lot of interest in them. However, neural networks typically require large datasets to approximate well the desired but unknown function, it can be computationally expensive to train or perform inference with them, and there are other limitations (see <a href="https://ai.stackexchange.com/a/18578/2444">this</a>), so neural networks are definitely not perfect tools, and there is the need to improve them to make them more efficient and useful in certain scenarios (e.g. scenarios where uncertainty estimation is required).</p>
  <p>I am not really familiar with research on polynomial regression, but it is possible that this and other tools have been overlooked by the ML community. You may want to have a look at <a href="https://arxiv.org/abs/1806.06850" rel="nofollow noreferrer">this paper</a>, which states that NNs are essentially doing polynomial regression, though I have not read it, so I don't know the details about the main ideas and results in this paper.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-30T17:02:05.430Z
Answers.LastActivityDate: 2021-01-21T23:38:42.803Z
Answers.LastEditDate: 2021-01-21T23:38:42.803Z
Answers.UserDisplayName: nbro
Body: <p>Consider a neural network, e.g. as presented by Nielsen <a href="http://neuralnetworksanddeeplearning.com" rel="nofollow noreferrer">here</a>. Abstractly, we just construct some function <span class="math-container">$f: \mathbb{R}^n \to [0,1]^m$</span> for some <span class="math-container">$n,m \in \mathbb{N}$</span> (i.e. the dimensions of the input and output space) that depends on a large set of parameters, <span class="math-container">$p_j$</span>. We then just define the cost function <span class="math-container">$C$</span> and calculate <span class="math-container">$\nabla_p C$</span> and just map <span class="math-container">$p \to p - \epsilon \nabla_p C$</span> repeatedly.</p>
  <p>The question is why do we choose <span class="math-container">$f$</span> to be what it is in standard neural networks, e.g. a bunch of linear combinations and sigmoids? One answer is that there a theorem saying any suitably nice function can be approximated using neural networks. But the same is true of other types of functions <span class="math-container">$f$</span>. The Stone-Weierstrass theorem gives that we could use polynomials in <span class="math-container">$n$</span> variables: <span class="math-container">$$f(x) =  c^0_0 + (c^1_1 x_1 + c^1_2 x_2 + \cdots + c^1_n x_n) + (c^2_{11}x_1 x_1 + c^2_{12} x_1x_2 + \cdots + c^2_{1n} x_1 x_2 + c^2_{21} x_2x_1 + c^2_{22} x_2x_2 + \cdots) + \cdots,$$</span></p>
  <p>and still have a nice approximation theorem. Here the gradient would be even easier to calculate. Why not use polynomials?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-08-30T02:34:47.193Z
Comments.Text: Here is a related question [What are the differences between artificial neural networks and other function approximators?](https://ai.stackexchange.com/q/21012/2444).
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><comparison><function-approximation>
Title: Why are neural networks preferred to other classification functions optimized by gradient decent
UserDisplayName: Physical Mathematics
ThreadText: # Why are neural networks preferred to other classification functions optimized by gradient decent
  
  (Asked by: username_0 on 2020-08-29)
  
  Consider a neural network, e.g. as presented by Nielsen [here](http://neuralnetworksanddeeplearning.com). Abstractly, we just construct some function $f: \mathbb{R}^n \to [0,1]^m$ for some $n,m \in \mathbb{N}$ (i.e. the dimensions of the input and output space) that depends on a large set of parameters, $p\_j$. We then just define the cost function $C$ and calculate $\nabla\_p C$ and just map $p \to p - \epsilon \nabla\_p C$ repeatedly.
  
  The question is why do we choose $f$ to be what it is in standard neural networks, e.g. a bunch of linear combinations and sigmoids? One answer is that there a theorem saying any suitably nice function can be approximated using neural networks. But the same is true of other types of functions $f$. The Stone-Weierstrass theorem gives that we could use polynomials in $n$ variables: $$f(x) = c^0\_0 + (c^1\_1 x\_1 + c^1\_2 x\_2 + \cdots + c^1\_n x\_n) + (c^2\_{11}x\_1 x\_1 + c^2\_{12} x\_1x\_2 + \cdots + c^2\_{1n} x\_1 x\_2 + c^2\_{21} x\_2x\_1 + c^2\_{22} x\_2x\_2 + \cdots) + \cdots,$$
  
  and still have a nice approximation theorem. Here the gradient would be even easier to calculate. Why not use polynomials?
  
  **Tags:** neural-networks, machine-learning, comparison, function-approximation
  
  > **Comments:**
  > **username_1**: Here is a related question [What are the differences between artificial neural networks and other function approximators?](https://ai.stackexchange.com/q/21012/2444).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-08-30. Score: 3 | Accepted answer
  
  You can indeed fit a polynomial to your labelled data, which is known as [polynomial regression](https://en.wikipedia.org/wiki/Polynomial_regression) (which can e.g. be done with the function [`numpy.polyfit`](https://numpy.org/doc/stable/reference/generated/numpy.polyfit.html)). One apparent limitation of polynomial regression is that, in practice, you need to assume that your data follows some specific polynomial of some degree $n$, i.e. you assume that your data has the form of the polynomial that you choose, which may not be true.
  
  When you use a neural network to solve a classification or regression problem, you also need to choose the activation functions, the number of neurons, how they are connected, etc., so you also need to limit the number and type of functions that you can learn with neural networks, i.e. the *hypothesis space*.
  
  Now, it is not necessarily a bad thing to limit the hypothesis space. In fact, *learning* is generally an *ill-posed problem*, i.e. in simple terms, there could be multiple solutions or no solutions at all ([and other problems](https://www.mit.edu/%7E9.520/spring12/slides/class02/class02.pdf)), so, actually, you often need to limit the hypothesis space to find some useful solutions (e.g. solutions that generalise better to unseen data). *Regularisations techniques* are ways of constraining the learning problem, and the hypothesis space (i.e. the set of functions that your learning algorithm can choose from), and thus making the learning problem well-posed.
  
  Neural networks are not preferred over polynomial regression because they are theoretically more powerful. In fact, both can approximate any continuous function [[1](https://link.springer.com/content/pdf/10.1007/BF02551274.pdf)], but these are just theoretical results, i.e. these results do not give you the magical formula to choose the most appropriate neural network or polynomial that best approximates the desired unknown function.
  
  In practice, neural networks have been proven to **effectively solve many tasks** (e.g. translation of natural language, playing go or atari games, image classification, etc.), so I would say that this is the main reason they are widely studied and there is a lot of interest in them. However, neural networks typically require large datasets to approximate well the desired but unknown function, it can be computationally expensive to train or perform inference with them, and there are other limitations (see [this](https://ai.stackexchange.com/a/18578/2444)), so neural networks are definitely not perfect tools, and there is the need to improve them to make them more efficient and useful in certain scenarios (e.g. scenarios where uncertainty estimation is required).
  
  I am not really familiar with research on polynomial regression, but it is possible that this and other tools have been overlooked by the ML community. You may want to have a look at [this paper](https://arxiv.org/abs/1806.06850), which states that NNs are essentially doing polynomial regression, though I have not read it, so I don't know the details about the main ideas and results in this paper.
  
  
  ---
  
  