Answers.Body: <p>It is correct to say that a sigmoid activation function would only work well as a model if the desired output is close to the sigmoid function applied to the input.  This is a trivial fact that applies to a single layer perceptron.  This is true for for the single layer case for any activation function, also a trivial fact.</p>
  
  <p>When the layer number is between one and infinity (two or more), the theory bifurcates.  The identity function becomes a special case:  Any number of layers that conform to a first degree polynomial, <span class="math-container">$ax + b$</span>, can be replaced with a single identity layers.  The alternative case, where there are multiple layers that functionally do not conform to a first degree polynomial, <span class="math-container">$ax + b$</span>, cannot be replaced by a single layer of some equally simple function.  The complexity increases geometrically, which is the entire point of multilayer perceptrons.</p>
  
  <p>Under particular constraints, the multilayer perceptron can produce a wide variety of functional behaviors that do not resemble the activation functions of the layers.</p>
  
  <p>For instance, a properly trained network using sigmoid activation functions, with sufficient layer depth and sufficient massive allocation of computing resources, could theoretically approximate the topography of the Himalayas.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-13T15:22:44.957Z
Answers.LastActivityDate: 2019-01-07T20:14:20.573Z
Answers.LastEditDate: 2019-01-07T20:14:20.573Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>Because any number of linear layers can be represented by a single linear layer:</p>
  
  <p><span class="math-container">$A_2*(A_1*x +b_1)+B_2 = A_2*A_1*x+ A2*b_1+b_2$</span></p>
  
  <p>The same is not true if you have non-linear functions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-13T17:14:07.473Z
Answers.LastActivityDate: 2018-10-13T17:14:07.473Z
Answers.UserDisplayName: Martin Thoma
Body: <p>A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why doesn't sigmoid only work when the relation between input and output is "of sigmoid shape"?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-08-24T23:50:54.203Z
Comments.Text: It should, but the chance of data being related by sigmoid is thin. Also, sigmoid function doesn't map data to the an 's' shape- it squeezes data between 0 and 1.
Comments.UserDisplayName: Daniel
Tags: <activation-functions><sigmoid>
Title: Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?
UserDisplayName: Mr. Eivind
ThreadText: # Why do non-linear activation functions not require a specific non-linear relation between its inputs and outputs?
  
  (Asked by: username_0 on 2018-08-23)
  
  A linear activation function (or none at all) should only be used when the relation between input and output is linear. Why doesn't the same rule apply for other activation functions? For example, why doesn't sigmoid only work when the relation between input and output is "of sigmoid shape"?
  
  **Tags:** activation-functions, sigmoid
  
  > **Comments:**
  > **username_1**: It should, but the chance of data being related by sigmoid is thin. Also, sigmoid function doesn't map data to the an 's' shape- it squeezes data between 0 and 1.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2018-10-13. Score: 1
  
  It is correct to say that a sigmoid activation function would only work well as a model if the desired output is close to the sigmoid function applied to the input. This is a trivial fact that applies to a single layer perceptron. This is true for for the single layer case for any activation function, also a trivial fact.
  
  When the layer number is between one and infinity (two or more), the theory bifurcates. The identity function becomes a special case: Any number of layers that conform to a first degree polynomial, $ax + b$, can be replaced with a single identity layers. The alternative case, where there are multiple layers that functionally do not conform to a first degree polynomial, $ax + b$, cannot be replaced by a single layer of some equally simple function. The complexity increases geometrically, which is the entire point of multilayer perceptrons.
  
  Under particular constraints, the multilayer perceptron can produce a wide variety of functional behaviors that do not resemble the activation functions of the layers.
  
  For instance, a properly trained network using sigmoid activation functions, with sufficient layer depth and sufficient massive allocation of computing resources, could theoretically approximate the topography of the Himalayas.
  
  
  ---
  
  ### Answer by username_3 on 2018-10-13. Score: -1
  
  Because any number of linear layers can be represented by a single linear layer:
  
  $A\_2\*(A\_1\*x +b\_1)+B\_2 = A\_2\*A\_1\*x+ A2\*b\_1+b\_2$
  
  The same is not true if you have non-linear functions.
  
  
  ---
  
  