Answers.Body: <p>Your analogy is correct, except it is not really an &quot;analogy&quot;. <a href="https://openreview.net/forum?id=Sks3zF9eg" rel="nofollow noreferrer">Sin is an activation function</a> - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.</p>
  <p>So your expression <span class="math-container">$\sigma(x) = A\sin ax + B \sin bx + D \sin ex$</span> is of a neural network with one 3-neuron layer and a single output linear neuron:</p>
  <p><span class="math-container">$$\sigma(x) = \sum_i V_i \sin\left(\sum_jW_{ij} x_j + b_i\right)+\beta$$</span></p>
  <p>With all biases being zero <span class="math-container">$b_i=\beta=0$</span>, output weights are <span class="math-container">$V_i = \left(A,B,C\right)$</span> and the inner weight matrix is diagonal:
  <span class="math-container">$$W_{ij} = \begin{pmatrix}a&amp;0&amp;0\\0&amp;b&amp;0\\0&amp;0&amp;e\end{pmatrix}$$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T22:38:08.620Z
Answers.Text: What made the non-periodic functions gain favor over sin?
Answers.UserDisplayName: sangstar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T21:40:28.573Z
Answers.LastActivityDate: 2021-04-15T21:40:28.573Z
Answers.UserDisplayName: Kostya
Answers.Body: <p>Ok. Here is an analogy for you. The equation for a neuron is wx + b, which is equivalent to a straight line. If we don't apply non-linearity we will be stuck with a straight line forever. So, this type of network won't be even able to model points in a unit circle randomly distributed.</p>
  <p>What does non-linearity do? If you look the graphs for x to the power 2, 3, 4 and so on. You see with each increase in power, the line gets tugged like a sine or cosine curve. That bend in the straight line allows us to then model boundaries with arbitrary shapes.</p>
  <p>The more the difficult the boundary to model between the classes the more bends in the line you need to model and so, you keep on increasing the layers in neural network.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-15T20:03:32.477Z
Answers.LastActivityDate: 2021-04-15T20:03:32.477Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are better at approximating a function than combinations of linear functions?</p>
  <p>For example, regardless of the amount of terms in this combination of linear functions, the function <em>will always</em> be some form of <span class="math-container">$y = mx + b$</span>. However, if we're summing sines, you <em>absolutely cannot</em> express a combination of sines and cosines as something of the form <span class="math-container">$A \sin{bx}$</span>. For example, a combination of three sinusoids cannot be simplified further than <span class="math-container">$A \sin{bx} + B \sin{cx} + D \sin{ex}$</span>.</p>
  <p>Is this fact essentially <em>why</em> the Fourier series is able to approximate functions (other than obviously the fact that <span class="math-container">$A \sin{bx}$</span> is orthogonal to <span class="math-container">$B \sin{cx}$</span>)? Because if it could be simplified into <em>one</em> sinusoid, it could never approximate an arbitrary function because it's lost its robustness? Because with other terms combined, whereas linear functions summed up gain no further ability to approximate, things like sinusoids actually begin to approximate really well with enough terms and with the right constants.</p>
  <p>In that vein, is this the reason why <em>non</em>-linear activiation functions (also called non-linear classifiers?) are generally valued more than linear ones? Because linear activation functions simply are lousy function approximators, while, with enough constants and terms, non-linear activation functions can approximate <em>any</em> function?</p>
  
Tags: <activation-functions><function-approximation>
Title: Trying to understand why nonlinearity is important for neural networks by analogy
UserDisplayName: sangstar
ThreadText: # Trying to understand why nonlinearity is important for neural networks by analogy
  
  (Asked by: username_0 on 2021-04-15)
  
  Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are better at approximating a function than combinations of linear functions?
  
  For example, regardless of the amount of terms in this combination of linear functions, the function *will always* be some form of $y = mx + b$. However, if we're summing sines, you *absolutely cannot* express a combination of sines and cosines as something of the form $A \sin{bx}$. For example, a combination of three sinusoids cannot be simplified further than $A \sin{bx} + B \sin{cx} + D \sin{ex}$.
  
  Is this fact essentially *why* the Fourier series is able to approximate functions (other than obviously the fact that $A \sin{bx}$ is orthogonal to $B \sin{cx}$)? Because if it could be simplified into *one* sinusoid, it could never approximate an arbitrary function because it's lost its robustness? Because with other terms combined, whereas linear functions summed up gain no further ability to approximate, things like sinusoids actually begin to approximate really well with enough terms and with the right constants.
  
  In that vein, is this the reason why *non*-linear activiation functions (also called non-linear classifiers?) are generally valued more than linear ones? Because linear activation functions simply are lousy function approximators, while, with enough constants and terms, non-linear activation functions can approximate *any* function?
  
  **Tags:** activation-functions, function-approximation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-04-15. Score: 2
  
  Your analogy is correct, except it is not really an "analogy". [Sin is an activation function](https://openreview.net/forum?id=Sks3zF9eg) - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.
  
  So your expression $\sigma(x) = A\sin ax + B \sin bx + D \sin ex$ is of a neural network with one 3-neuron layer and a single output linear neuron:
  
  $$\sigma(x) = \sum\_i V\_i \sin\left(\sum\_jW\_{ij} x\_j + b\_i\right)+\beta$$
  
  With all biases being zero $b\_i=\beta=0$, output weights are $V\_i = \left(A,B,C\right)$ and the inner weight matrix is diagonal:
  $$W\_{ij} = \begin{pmatrix}a&0&0\\0&b&0\\0&0&e\end{pmatrix}$$
  
  > **Comments:**
  > **username_0**: What made the non-periodic functions gain favor over sin?  
  
  ---
  
  ### Answer by username_2 on 2021-04-15. Score: 0
  
  Ok. Here is an analogy for you. The equation for a neuron is wx + b, which is equivalent to a straight line. If we don't apply non-linearity we will be stuck with a straight line forever. So, this type of network won't be even able to model points in a unit circle randomly distributed.
  
  What does non-linearity do? If you look the graphs for x to the power 2, 3, 4 and so on. You see with each increase in power, the line gets tugged like a sine or cosine curve. That bend in the straight line allows us to then model boundaries with arbitrary shapes.
  
  The more the difficult the boundary to model between the classes the more bends in the line you need to model and so, you keep on increasing the layers in neural network.
  
  
  ---
  
  