Answers.Body: <p>First of all, neural networks are not (just) defined by the fact that they are typically trained with gradient descent and back-propagation. In fact, there are other ways of training neural networks, such as evolutionary algorithms and the Hebb's rule (e.g. Hopfield networks are typically associated with this Hebbian learning rule). </p>
  
  <p>The first difference between neural networks and other function approximators is <strong>conceptual</strong>. In neural networks, you typically imagine that there are one or more computational units (often called <em>neurons</em>) that are connected in different and often complex ways. The human can choose these connections (or they could also be learned) and the functions that these units compute given the inputs. So, there's a great deal of <strong>flexibility</strong> and complexity, but, often, also a lack of rigorousness (from the mathematical point of view) while using and designing neuron networks. </p>
  
  <p>The other difference is that neural networks were originally inspired by the biological counterparts. See <a href="https://link.springer.com/article/10.1007/BF02478259" rel="nofollow noreferrer">A logical calculus of the ideas immanent in nervous activity</a> (1943) by Warren McCulloch and Walter Pitts, who proposed, inspired by neuroscience, the first mathematical model of an artificial neuron.</p>
  
  <p>There are other technical differences. For example, the <a href="https://mathworld.wolfram.com/TaylorSeries.html" rel="nofollow noreferrer">Taylor expansion of a function</a> is <em>typically</em> done only at a single value of the domain, it assumes that the function to be approximated is differentiable multiple times, and it makes uses of the derivatives of such a function. Fourier series typically approximate functions with a weighted sum of sinusoids. Given appropriate weights, the <a href="https://en.wikipedia.org/wiki/Fourier_series" rel="nofollow noreferrer">Fourier series</a> can be used to approximate an arbitrary function in a certain interval or the entire function (if the function you want to approximate is also periodic). On the other hand, neural networks attempt to approximate functions of the form <span class="math-container">$f: [0, 1]^n \rightarrow \mathbb{R}$</span> (at least, this is the setup in <a href="https://link.springer.com/article/10.1007/BF02551274" rel="nofollow noreferrer">the famous paper that proved the universality of neural networks</a>) in many different ways (for example, weighted sums followed by sigmoids).</p>
  
  <p>To conclude, neural networks are quite different from other function approximation techniques (such as Taylor or Fourier series) in the way they approximate functions and their purpose (i.e. which functions they were supposed to approximate and in which context).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-09T14:48:29.760Z
Answers.LastActivityDate: 2020-05-16T22:56:13.537Z
Answers.LastEditDate: 2020-05-16T22:56:13.537Z
Answers.UserDisplayName: nbro
Body: <p>Modern artificial neural networks use a lot more functions than just the classic sigmoid, to the point I'm having a hard time really seeing what classifies something as a "neural network" over other function approximators (such as Fourier series, Bernstein polynomials, Chebyshev polynomials or splines). </p>
  
  <p>So, what makes something an artificial neural network? Is there a subset of theorems that apply only to neural networks? </p>
  
  <p>Backpropagation is classic, but that is the multi-variable chain rule, what else is unique to neural networks over other function approximators? </p>
  
Tags: <neural-networks><machine-learning><comparison><function-approximation>
Title: What are the differences between artificial neural networks and other function approximators?
UserDisplayName: FourierFlux
ThreadText: # What are the differences between artificial neural networks and other function approximators?
  
  (Asked by: username_0 on 2020-05-08)
  
  Modern artificial neural networks use a lot more functions than just the classic sigmoid, to the point I'm having a hard time really seeing what classifies something as a "neural network" over other function approximators (such as Fourier series, Bernstein polynomials, Chebyshev polynomials or splines).
  
  So, what makes something an artificial neural network? Is there a subset of theorems that apply only to neural networks?
  
  Backpropagation is classic, but that is the multi-variable chain rule, what else is unique to neural networks over other function approximators?
  
  **Tags:** neural-networks, machine-learning, comparison, function-approximation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-09. Score: 2
  
  First of all, neural networks are not (just) defined by the fact that they are typically trained with gradient descent and back-propagation. In fact, there are other ways of training neural networks, such as evolutionary algorithms and the Hebb's rule (e.g. Hopfield networks are typically associated with this Hebbian learning rule).
  
  The first difference between neural networks and other function approximators is **conceptual**. In neural networks, you typically imagine that there are one or more computational units (often called *neurons*) that are connected in different and often complex ways. The human can choose these connections (or they could also be learned) and the functions that these units compute given the inputs. So, there's a great deal of **flexibility** and complexity, but, often, also a lack of rigorousness (from the mathematical point of view) while using and designing neuron networks.
  
  The other difference is that neural networks were originally inspired by the biological counterparts. See [A logical calculus of the ideas immanent in nervous activity](https://link.springer.com/article/10.1007/BF02478259) (1943) by Warren McCulloch and Walter Pitts, who proposed, inspired by neuroscience, the first mathematical model of an artificial neuron.
  
  There are other technical differences. For example, the [Taylor expansion of a function](https://mathworld.wolfram.com/TaylorSeries.html) is *typically* done only at a single value of the domain, it assumes that the function to be approximated is differentiable multiple times, and it makes uses of the derivatives of such a function. Fourier series typically approximate functions with a weighted sum of sinusoids. Given appropriate weights, the [Fourier series](https://en.wikipedia.org/wiki/Fourier_series) can be used to approximate an arbitrary function in a certain interval or the entire function (if the function you want to approximate is also periodic). On the other hand, neural networks attempt to approximate functions of the form $f: [0, 1]^n \rightarrow \mathbb{R}$ (at least, this is the setup in [the famous paper that proved the universality of neural networks](https://link.springer.com/article/10.1007/BF02551274)) in many different ways (for example, weighted sums followed by sigmoids).
  
  To conclude, neural networks are quite different from other function approximation techniques (such as Taylor or Fourier series) in the way they approximate functions and their purpose (i.e. which functions they were supposed to approximate and in which context).
  
  
  ---
  
  