Answers.Body: <p>The problem with certain activation functions, such as the sigmoid, is that they squash the input to a finite interval (i.e. they are sometimes classified as <em>saturating activation functions</em>). For example, the sigmoid function has codomain <span class="math-container">$[0, 1]$</span>, as you can see from the illustration below.</p>
  <p><a href="https://i.sstatic.net/Fhg3s.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Fhg3s.png" alt="enter image description here" /></a></p>
  <p>This property/behaviour can lead to the vanishing gradient problem (which was one of the problems that Sepp Hochreiter, the author of the LSTM, was trying to solve in the context of recurrent neural networks, when developing the LSTM, along with his advisor, Schmidhuber).</p>
  <p>Empirically, people have noticed that ReLU can avoid this vanishing gradient problem. See e.g. <a href="https://machinelearningmastery.com/how-to-fix-vanishing-gradients-using-the-rectified-linear-activation-function/" rel="nofollow noreferrer">this blog post</a>. The paper <a href="http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf" rel="nofollow noreferrer">Deep Sparse Rectifier Neural Networks</a> provides more details about the advantage of ReLUs (aka rectifiers), so you may want to read it. However, ReLUs can also suffer from another (opposite) problem, i.e. the exploding gradient problem. Nevertheless, there are several ways to combat this issue. See e.g. <a href="https://machinelearningmastery.com/exploding-gradients-in-neural-networks/" rel="nofollow noreferrer">this blog post</a>.</p>
  <p>That being said, I am not an expert on residual networks, but I think that they used the ReLU to further avoid the vanishing gradient problem. <a href="https://ai.stackexchange.com/a/17785/2444">This answer</a> (that I gave some time ago) should give you some intuition about why residual networks can avoid the vanishing gradient problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-25T03:32:36.850Z
Answers.LastActivityDate: 2020-07-25T03:38:16.443Z
Answers.LastEditDate: 2020-07-25T03:38:16.443Z
Answers.UserDisplayName: nbro
Body: <p>In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic tangent, etc.?</p>
  <p><a href="https://i.sstatic.net/G42j3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/G42j3.png" alt="enter image description here" /></a></p>
  
Tags: <neural-networks><convolutional-neural-networks><activation-functions><relu><residual-networks>
Title: Can residual neural networks use other activation functions different from ReLU?
UserDisplayName: jr123456jr987654321
ThreadText: # Can residual neural networks use other activation functions different from ReLU?
  
  (Asked by: username_0 on 2020-07-23)
  
  In many diagrams, as seen below, residual neural networks are only depicted with ReLU activation functions, but can residual NNs also use other activation functions, such as the sigmoid, hyperbolic tangent, etc.?
  
  [![enter image description here](https://i.sstatic.net/G42j3.png)](https://i.sstatic.net/G42j3.png)
  
  **Tags:** neural-networks, convolutional-neural-networks, activation-functions, relu, residual-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-25. Score: 2 | Accepted answer
  
  The problem with certain activation functions, such as the sigmoid, is that they squash the input to a finite interval (i.e. they are sometimes classified as *saturating activation functions*). For example, the sigmoid function has codomain $[0, 1]$, as you can see from the illustration below.
  
  [![enter image description here](https://i.sstatic.net/Fhg3s.png)](https://i.sstatic.net/Fhg3s.png)
  
  This property/behaviour can lead to the vanishing gradient problem (which was one of the problems that Sepp Hochreiter, the author of the LSTM, was trying to solve in the context of recurrent neural networks, when developing the LSTM, along with his advisor, Schmidhuber).
  
  Empirically, people have noticed that ReLU can avoid this vanishing gradient problem. See e.g. [this blog post](https://machinelearningmastery.com/how-to-fix-vanishing-gradients-using-the-rectified-linear-activation-function/). The paper [Deep Sparse Rectifier Neural Networks](http://proceedings.mlr.press/v15/glorot11a/glorot11a.pdf) provides more details about the advantage of ReLUs (aka rectifiers), so you may want to read it. However, ReLUs can also suffer from another (opposite) problem, i.e. the exploding gradient problem. Nevertheless, there are several ways to combat this issue. See e.g. [this blog post](https://machinelearningmastery.com/exploding-gradients-in-neural-networks/).
  
  That being said, I am not an expert on residual networks, but I think that they used the ReLU to further avoid the vanishing gradient problem. [This answer](https://ai.stackexchange.com/a/17785/2444) (that I gave some time ago) should give you some intuition about why residual networks can avoid the vanishing gradient problem.
  
  
  ---
  
  