Answers.Body: <p>Not sure this is answering your question but, one way to get a general idea of the number of neurons needed is to consider the number of turning points in the polynomial.</p>
  <p>Each turning point in a polynomial corresponds to a change in the sign of its derivative, and the more turning points a polynomial has, the more complex its shape. For a polynomial of degree n, there can be at most n-1 turning points. This means that a polynomial with many turning points will require more neurons in the hidden layer to be accurately approximated.</p>
  <p>In general, increasing the number of neurons in the hidden layer can improve the model's ability to express a polynomial. Adding one more neuron to the hidden layer can allow the network to capture more complex patterns in the data, which can lead to a more accurate approximation of the polynomial.</p>
  <p>I have found these two sources very useful in visualising the above. Hope it helps.</p>
  <p><a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="nofollow noreferrer">http://neuralnetworksanddeeplearning.com/chap4.html</a>
  <a href="https://cs.stanford.edu/people/karpathy/convnetjs/demo/regression.html" rel="nofollow noreferrer">https://cs.stanford.edu/people/karpathy/convnetjs/demo/regression.html</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-20T12:23:46.763Z
Answers.LastActivityDate: 2023-04-20T12:23:46.763Z
Answers.UserDisplayName: hH1sG0n3
Body: <p>I learned about the <em>universal approximation theorem</em> from <a href="http://neuralnetworksanddeeplearning.com/chap4.html" rel="noreferrer">this</a> guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number of neurons. Or mathematically, <span class="math-container">${|g(x)−f(x)|&lt; \epsilon}$</span>, where <span class="math-container">${g(x)}$</span> is the approximation, <span class="math-container">${f(x)}$</span> is the target function and  is <span class="math-container">$\epsilon$</span> is an arbitrary bound.</p>
  <p>A polynomial of degree <span class="math-container">$n$</span> has at maximum <span class="math-container">$n-1$</span> turning points (where the derivative of the polynomial changes sign). With each new turning point, the approximation seems to become more complex.</p>
  <p>I'm not necessarily looking for a formula, but I'd like to get a general idea on how to figure out the sufficient number of neurons is for a reasonable approximation of a polynomial with a single layer of the neural network (you may consider &quot;reasonable&quot; to be <span class="math-container">$\epsilon = 0.0001$</span>). To ask in other words, how would adding one more neuron affect the model's ability to express a polynomial?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-04T15:38:58.797Z
Comments.Text: If the network has one hidden layer, there are papers like [A closer look at the approximation capabilities of neural networks (Kai Fong Ernest Chong)](https://openreview.net/forum?id=rkevSgrtPr) saying *"... given approximation threshold $\epsilon$ ... if $X$ in $\mathbb R^n$ is compact, then a neural network with $n$ input units, $m$ output units, and a single hidden layer with $\binom{n+d}{d}$ hidden units (independent of $m$ and $\epsilon$), can uniformly approximate any polynomial function $f:X \to \mathbb R^m$ whose total degree is at most $d$ for each of its $m$ coordinate functions."*
Comments.UserDisplayName: Vepir
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-16T19:24:06.153Z
Comments.Text: @Vepir - that sounds much more like an answer than a comment.
Comments.UserDisplayName: EngrStudent
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-15T01:05:32.580Z
Comments.Text: Not an answer, but I find that [simple neural networks are quite expressive](https://galenseilis.github.io/posts/nn-on-elementary-functions/). I suspect that any complete answer will have to address the network topology in some way.
Comments.UserDisplayName: Galen
Tags: <neural-networks><computational-learning-theory><multilayer-perceptrons><function-approximation><universal-approximation-theorems>
Title: What is the number of neurons required to approximate a polynomial of degree n?
UserDisplayName: mark mark
ThreadText: # What is the number of neurons required to approximate a polynomial of degree n?
  
  (Asked by: username_0 on 2021-01-02)
  
  I learned about the *universal approximation theorem* from [this](http://neuralnetworksanddeeplearning.com/chap4.html) guide. It states that a network even with a single hidden layer can approximate any function within some bound, given a sufficient number of neurons. Or mathematically, ${|g(x)−f(x)|< \epsilon}$, where ${g(x)}$ is the approximation, ${f(x)}$ is the target function and is $\epsilon$ is an arbitrary bound.
  
  A polynomial of degree $n$ has at maximum $n-1$ turning points (where the derivative of the polynomial changes sign). With each new turning point, the approximation seems to become more complex.
  
  I'm not necessarily looking for a formula, but I'd like to get a general idea on how to figure out the sufficient number of neurons is for a reasonable approximation of a polynomial with a single layer of the neural network (you may consider "reasonable" to be $\epsilon = 0.0001$). To ask in other words, how would adding one more neuron affect the model's ability to express a polynomial?
  
  **Tags:** neural-networks, computational-learning-theory, multilayer-perceptrons, function-approximation, universal-approximation-theorems
  > **Comments:**
  > **username_1**: If the network has one hidden layer, there are papers like [A closer look at the approximation capabilities of neural networks (Kai Fong Ernest Chong)](https://openreview.net/forum?id=rkevSgrtPr) saying *"... given approximation threshold $\epsilon$ ... if $X$ in $\mathbb R^n$ is compact, then a neural network with $n$ input units, $m$ output units, and a single hidden layer with $\binom{n+d}{d}$ hidden units (independent of $m$ and $\epsilon$), can uniformly approximate any polynomial function $f:X \to \mathbb R^m$ whose total degree is at most $d$ for each of its $m$ coordinate functions."*  
  > **username_2**: @Vepir - that sounds much more like an answer than a comment.  
  > **username_3**: Not an answer, but I find that [simple neural networks are quite expressive](https://galenseilis.github.io/posts/nn-on-elementary-functions/). I suspect that any complete answer will have to address the network topology in some way.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_4 on 2023-04-20. Score: 2
  
  Not sure this is answering your question but, one way to get a general idea of the number of neurons needed is to consider the number of turning points in the polynomial.
  
  Each turning point in a polynomial corresponds to a change in the sign of its derivative, and the more turning points a polynomial has, the more complex its shape. For a polynomial of degree n, there can be at most n-1 turning points. This means that a polynomial with many turning points will require more neurons in the hidden layer to be accurately approximated.
  
  In general, increasing the number of neurons in the hidden layer can improve the model's ability to express a polynomial. Adding one more neuron to the hidden layer can allow the network to capture more complex patterns in the data, which can lead to a more accurate approximation of the polynomial.
  
  I have found these two sources very useful in visualising the above. Hope it helps.
  
  <http://neuralnetworksanddeeplearning.com/chap4.html>
  <https://cs.stanford.edu/people/karpathy/convnetjs/demo/regression.html>
  
  
  ---
  
  