Answers.Body: <p>Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs the derivatives of all the activation functions and thus possibly leads to unbounded gradient if <em>any</em> of these are mere polynomials.</p>
  <p>And from the abstract of your referenced paper <em>threshold</em> plays an important role in universal function approximator, an obvious easy example is the 2 layer perceptrons (aka threshold logic unit [TLU], Adaline [Widrow, 1962]) neural network implementation of the classic Boolean even-parity function <span class="math-container">$y=x_1x_2+\bar x_1 \bar x_2$</span> which is not <a href="https://en.wikipedia.org/wiki/Linear_separability" rel="nofollow noreferrer">linearly separable</a> and thus a single layer of perceptrons cannot achieve as pointed out by Minsky in the 60's，but it could be easily solved by below 2 layer perceptrons with all (bounded) step functions assigned with appropriate thresholds as shown below (you can manually try to verify). If all activation functions are mere polynomials, no matter how many hidden layers you add, these Boolean functions cannot be modeled exactly, and, thus, most practical classification problems could not be solved by neural networks.</p>
  <p><a href="https://i.sstatic.net/b0KlF.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/b0KlF.png" alt="enter image description here" /></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T16:17:56.370Z
Answers.Text: Do you think you can provide a simple example why a polynomial nonlinearity would lead to an unbounded gradient? I don't see how that could be the case. (The derivative of a polynomial is just a polynomial, not infinity)
Answers.UserDisplayName: Steven Sagona
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T18:01:45.073Z
Answers.Text: @StevenSagona Here *unbounded* just means without a pre-assigned fixed threshold, (higher order) polynomial activation functions may contain bilinear terms which could complicate and amplify the (partial) derivatives. For example see [this paper](https://www.sciencedirect.com/science/article/abs/pii/S0950705121005207) about tensor network: *the mathematical explanation of neural tensor networks remains a challenging problem, due to the bilinear term... we associate NTN with Taylor’s theorem and find that each slice of NTN could be represented as a 2nd order multivariate Taylor polynomial.*
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-22T06:30:46.043Z
Answers.LastActivityDate: 2022-11-22T14:47:27.737Z
Answers.LastEditDate: 2022-11-22T14:47:27.737Z
Answers.UserDisplayName: cinch
Body: <p>Can someone give me the main idea of the paper <a href="https://www.sciencedirect.com/science/article/abs/pii/S0893608005801315" rel="nofollow noreferrer">Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function</a>? I'm having trouble understanding it.</p>
  
Tags: <neural-networks><papers><activation-functions><universal-approximation-theorems>
Title: Why do activation functions in neural networks have to be non-polynomial to approximate any function?
UserDisplayName: Alicia Chi
ThreadText: # Why do activation functions in neural networks have to be non-polynomial to approximate any function?
  
  (Asked by: username_0 on 2022-11-19)
  
  Can someone give me the main idea of the paper [Multilayer Feedforward Networks With a Nonpolynomial Activation Function Can Approximate Any Function](https://www.sciencedirect.com/science/article/abs/pii/S0893608005801315)? I'm having trouble understanding it.
  
  **Tags:** neural-networks, papers, activation-functions, universal-approximation-theorems
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-22. Score: 3
  
  Polynomials are unbounded once the input variable is very large or negative, also most feedforward NNs are using backpropagation algorithms to adjust weights during each training iteration which needs the derivatives of all the activation functions and thus possibly leads to unbounded gradient if *any* of these are mere polynomials.
  
  And from the abstract of your referenced paper *threshold* plays an important role in universal function approximator, an obvious easy example is the 2 layer perceptrons (aka threshold logic unit [TLU], Adaline [Widrow, 1962]) neural network implementation of the classic Boolean even-parity function $y=x\_1x\_2+\bar x\_1 \bar x\_2$ which is not [linearly separable](https://en.wikipedia.org/wiki/Linear_separability) and thus a single layer of perceptrons cannot achieve as pointed out by Minsky in the 60's，but it could be easily solved by below 2 layer perceptrons with all (bounded) step functions assigned with appropriate thresholds as shown below (you can manually try to verify). If all activation functions are mere polynomials, no matter how many hidden layers you add, these Boolean functions cannot be modeled exactly, and, thus, most practical classification problems could not be solved by neural networks.
  
  [![enter image description here](https://i.sstatic.net/b0KlF.png)](https://i.sstatic.net/b0KlF.png)
  
  > **Comments:**
  > **username_2**: Do you think you can provide a simple example why a polynomial nonlinearity would lead to an unbounded gradient? I don't see how that could be the case. (The derivative of a polynomial is just a polynomial, not infinity)  
  > **username_1**: @StevenSagona Here *unbounded* just means without a pre-assigned fixed threshold, (higher order) polynomial activation functions may contain bilinear terms which could complicate and amplify the (partial) derivatives. For example see [this paper](https://www.sciencedirect.com/science/article/abs/pii/S0950705121005207) about tensor network: *the mathematical explanation of neural tensor networks remains a challenging problem, due to the bilinear term... we associate NTN with Taylor’s theorem and find that each slice of NTN could be represented as a 2nd order multivariate Taylor polynomial.*  
  
  ---
  
  