Answers.Body: <p>A multi-layer network in which all units have linear activation functions can always be collapsed to an equivalent network with two layers of units. That is why it is essential to use nonlinear unit activation functions. </p>
  
  <p>The underlying reason for using nonlinear activation functions involves a remarkable theorem of Cybenko (1989), which states that one layer of nonlinear hidden units is sufficient to approximate any mapping from input to output units. Actually, I think there is a later proof which specifies that the nonlinearity can be any non-polynomial function (e.g. sigmoidal).</p>
  
  <p>This text is based on the book: <a href="https://jim-stone.staff.shef.ac.uk/BookBayes2012/books_by_jv_stone/" rel="nofollow noreferrer">Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T13:19:54.097Z
Answers.Text: Links to the proof of Cybenko's theorem would be more useful to OP than links to your book, unless you can specify where in your book the topic is covered. Ideally your book outlines Cybenko's theorem?
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-24T10:37:01.143Z
Answers.Text: The Cybenko's proof assumes that a sigmoid is used (if I remember correctly), but it's true that other proofs have been provided that are more general.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-10T15:52:00.210Z
Answers.Text: i would be led to believe that the proof came *before* the practice of using non linear activation functions
Answers.UserDisplayName: k.c. sayz 'k.c sayz'
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-24T08:34:19.723Z
Answers.LastActivityDate: 2020-01-24T12:33:38.147Z
Answers.LastEditDate: 2020-01-24T12:33:38.147Z
Answers.UserDisplayName: James V Stone
Body: <p>The <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="noreferrer">ReLU</a> activation function is defined as follows </p>
  
  <p><span class="math-container">$$y = \operatorname{max}(0,x)$$</span></p>
  
  <p>And the linear activation function is defined as follows</p>
  
  <p><span class="math-container">$$y = x$$</span></p>
  
  <p>The ReLU nonlinearity just clips the values less than 0 to 0 and passes everything else. Then why not to use a linear activation function instead, as it will pass all the gradient information during backpropagation? I do see that parametric ReLU (PReLU) does provide this possibility. </p>
  
  <p>I just want to know if there is a proper explanation to using ReLU as default or it is just based on observations that it performs better on the training sets.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-05-19T18:00:41.460Z
Comments.Text: Use as interpretation of the activation function result the one of "error". Relu has a value 0, meaning expected optimal result is reached. Lineal has a minimum of less infinite, meaning the error can be less than zero (not applicable in most cases) and always posible to improve (near than never applicable)
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-02-18T10:30:04.480Z
Comments.Text: A simple intuition behind this, is that an ANN with all linear activations is analogous to linear regression
Comments.UserDisplayName: hisairnessag3
Tags: <neural-networks><deep-learning><comparison><activation-functions><relu>
Title: Why do we prefer ReLU over linear activation functions?
UserDisplayName: imflash217
ThreadText: # Why do we prefer ReLU over linear activation functions?
  
  (Asked by: username_0 on 2018-05-19)
  
  The [ReLU](https://en.wikipedia.org/wiki/Rectifier_(neural_networks)) activation function is defined as follows
  
  $$y = \operatorname{max}(0,x)$$
  
  And the linear activation function is defined as follows
  
  $$y = x$$
  
  The ReLU nonlinearity just clips the values less than 0 to 0 and passes everything else. Then why not to use a linear activation function instead, as it will pass all the gradient information during backpropagation? I do see that parametric ReLU (PReLU) does provide this possibility.
  
  I just want to know if there is a proper explanation to using ReLU as default or it is just based on observations that it performs better on the training sets.
  
  **Tags:** neural-networks, deep-learning, comparison, activation-functions, relu
  > **Comments:**
  > **username_1**: Use as interpretation of the activation function result the one of "error". Relu has a value 0, meaning expected optimal result is reached. Lineal has a minimum of less infinite, meaning the error can be less than zero (not applicable in most cases) and always posible to improve (near than never applicable)  
  > **username_2**: A simple intuition behind this, is that an ANN with all linear activations is analogous to linear regression  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2020-01-24. Score: 2
  
  A multi-layer network in which all units have linear activation functions can always be collapsed to an equivalent network with two layers of units. That is why it is essential to use nonlinear unit activation functions.
  
  The underlying reason for using nonlinear activation functions involves a remarkable theorem of Cybenko (1989), which states that one layer of nonlinear hidden units is sufficient to approximate any mapping from input to output units. Actually, I think there is a later proof which specifies that the nonlinearity can be any non-polynomial function (e.g. sigmoidal).
  
  This text is based on the book: [Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning](https://jim-stone.staff.shef.ac.uk/BookBayes2012/books_by_jv_stone/).
  
  > **Comments:**
  > **username_4**: Links to the proof of Cybenko's theorem would be more useful to OP than links to your book, unless you can specify where in your book the topic is covered. Ideally your book outlines Cybenko's theorem?  
  > **username_5**: The Cybenko's proof assumes that a sigmoid is used (if I remember correctly), but it's true that other proofs have been provided that are more general.  
  > **username_6**: i would be led to believe that the proof came *before* the practice of using non linear activation functions  
  
  ---
  
  