Answers.Body: <h3>tl;dr</h3>
  
  <p>Your intuition is correct.</p>
  
  <h3>Why is this a problem?</h3>
  
  <p>The <em>saturating</em> effect of the sigmoid activation function is well documented as it is the main culprit of a problem called <strong>vanishing gradients</strong>.</p>
  
  <p>In short, the derivative of the sigmoid function is:</p>
  
  <p><span class="math-container">$$
  \frac{dσ(z)}{dz} = σ(z) (1 - σ(z))
  $$</span></p>
  
  <p>The problem is that very often (especially if we initialize our weights with <strong>large values</strong>), the output of this neuron will either be <span class="math-container">$1$</span> or <span class="math-container">$0$</span>. This causes the gradient to be <span class="math-container">$0$</span>, which in turn means that the weights of that neuron can't be updated. </p>
  
  <h3>So let's keep the weights small (in absolute value)</h3>
  
  <p>Initially this seems like a good idea, if we keep the weights small we will avoid saturation.</p>
  
  <p><img src="https://proxy.duckduckgo.com/iu/?u=https%3A%2F%2Ftse1.mm.bing.net%2Fth%3Fid%3DOIP.HXCBO-Wx5XhuY_OwMl0PhwHaE8%26pid%3DApi&amp;f=1" alt=""></p>
  
  <p>However there is another problem with sigmoid functions: their gradient has a maximum value of <span class="math-container">$0.25$</span>! This means that if the weight of a neuron is less than <span class="math-container">$4$</span>, the error will <strong>diminish</strong> while flowing backward through the net. This becomes progressively worse as we add more layers to the network. </p>
  
  <h3>How have we solved this issue</h3>
  
  <p>Naturally researchers tried to find better weight initialization strategies. However this was hard, because as we saw, small weights are bad and large weights are bad. </p>
  
  <p>One example is <a href="https://arxiv.org/pdf/1511.06422.pdf" rel="nofollow noreferrer">Mishkin et al. 2016</a> who propose a new initialization strategy, but fail to train a deep neural network with sigmoid activations.</p>
  
  <p>Another workaround is to use a different learning rate for each layer (<a href="https://openreview.net/pdf?id=D1VDjyJjXF5jEJ1zfE53" rel="nofollow noreferrer">Xu et al. - Revise Saturated Activation Functions</a>)</p>
  
  <p>After a while the Machine Learning community realized that sigmoid functions were ill-suited for deep neural networks and adopted ReLU activations, which have fewer drawbacks and scale better. Nowadays, they have become the de-facto choice for deep learning.</p>
  
  <h3>Sources</h3>
  
  <p>This problem is known for several years and has been well documented (the earliest I could find was in 1994). This was mostly explored in Recurrent Neural Networks.</p>
  
  <p>If you are interested in reading about this, I'd recommend <a href="https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b" rel="nofollow noreferrer">this post</a>,  by Andrej Karpathy.</p>
  
  <p>Some more formal sources on this topic:</p>
  
  <ul>
  <li><a href="https://www.bioinf.jku.at/publications/older/ch7.pdf" rel="nofollow noreferrer">Hochreiter et al. - Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies</a></li>
  <li><a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.7389&amp;rep=rep1&amp;type=pdf" rel="nofollow noreferrer">Hochreiter et al. - Recurrent Neural Net Learning and Vanishing Gradient</a></li>
  <li><a href="http://proceedings.mlr.press/v28/pascanu13.pdf" rel="nofollow noreferrer">Pascanu et al. - On the difficulty of training recurrent neural networks</a></li>
  <li><a href="http://ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdf" rel="nofollow noreferrer">Bengio et al. - Learning long-term dependencies with gradient descent is difficult</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-24T23:32:56.393Z
Answers.LastActivityDate: 2019-06-24T23:32:56.393Z
Answers.UserDisplayName: Djib2011
Body: <p>Suppose I have a deep feed-forward neural network with sigmoid activation <span class="math-container">$\sigma$</span> already trained on a dataset <span class="math-container">$S$</span>. Let's consider a training point <span class="math-container">$x_i \in S$</span>. I want to analyze the entries of a hidden layer <span class="math-container">$h_{i,l}$</span>, where</p>
  
  <p><span class="math-container">$$h_{i,l} = \sigma(W_l ( \sigma (W_{l-1} \sigma( \dots \sigma ( W_1 \cdot x_i))\dots).        $$</span></p>
  
  <p>My intuition would be that, since gradient descend has passed many times on the point <span class="math-container">$x_i$</span> updating the weights at every iteration, the entries of every hidden layer computed on <span class="math-container">$x_i$</span> would be either very close to zero or very close to one (thanks to the effect of the sigmoid activation). </p>
  
  <p>Is this true? Is there a theoretical result in the literature which shows anything similar to this? Is there an empirical result which shows that?  </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-28T10:31:48.947Z
Comments.Text: Interesting question, I think Tensorflow and similar frameworks have mechanisms to take a look under the hood, so you might find answers with a relatively simple experiment. I'm not sure whether your intuition is right, but that's just my intuition :-)
Comments.UserDisplayName: Hans-Martin Mosner
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-06-25T05:03:59.497Z
Comments.Text: Weights in trained image classifier network have Gaussian distribution for each layer, with mean/var *not* close to 1, but that is my own anecdotal experience.  I think it was supported by 1 paper, but I don't recall its title. PS even if initial weight were not Gaussian
Comments.UserDisplayName: mirror2image
Tags: <neural-networks><machine-learning><deep-learning><activation-functions>
Title: How do intermediate layers of a trained neural network look like?
UserDisplayName: Alfred
ThreadText: # How do intermediate layers of a trained neural network look like?
  
  (Asked by: username_0 on 2019-03-27)
  
  Suppose I have a deep feed-forward neural network with sigmoid activation $\sigma$ already trained on a dataset $S$. Let's consider a training point $x\_i \in S$. I want to analyze the entries of a hidden layer $h\_{i,l}$, where
  
  $$h\_{i,l} = \sigma(W\_l ( \sigma (W\_{l-1} \sigma( \dots \sigma ( W\_1 \cdot x\_i))\dots). $$
  
  My intuition would be that, since gradient descend has passed many times on the point $x\_i$ updating the weights at every iteration, the entries of every hidden layer computed on $x\_i$ would be either very close to zero or very close to one (thanks to the effect of the sigmoid activation).
  
  Is this true? Is there a theoretical result in the literature which shows anything similar to this? Is there an empirical result which shows that?
  
  **Tags:** neural-networks, machine-learning, deep-learning, activation-functions
  
  > **Comments:**
  > **username_1**: Interesting question, I think Tensorflow and similar frameworks have mechanisms to take a look under the hood, so you might find answers with a relatively simple experiment. I'm not sure whether your intuition is right, but that's just my intuition :-)  
  > **username_2**: Weights in trained image classifier network have Gaussian distribution for each layer, with mean/var *not* close to 1, but that is my own anecdotal experience.  I think it was supported by 1 paper, but I don't recall its title. PS even if initial weight were not Gaussian  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2019-06-24. Score: 2 | Accepted answer
  
  ### tl;dr
  
  Your intuition is correct.
  
  ### Why is this a problem?
  
  The *saturating* effect of the sigmoid activation function is well documented as it is the main culprit of a problem called **vanishing gradients**.
  
  In short, the derivative of the sigmoid function is:
  
  $$
  \frac{dσ(z)}{dz} = σ(z) (1 - σ(z))
  $$
  
  The problem is that very often (especially if we initialize our weights with **large values**), the output of this neuron will either be $1$ or $0$. This causes the gradient to be $0$, which in turn means that the weights of that neuron can't be updated.
  
  ### So let's keep the weights small (in absolute value)
  
  Initially this seems like a good idea, if we keep the weights small we will avoid saturation.
  
  ![](https://proxy.duckduckgo.com/iu/?u=https%3A%2F%2Ftse1.mm.bing.net%2Fth%3Fid%3DOIP.HXCBO-Wx5XhuY_OwMl0PhwHaE8%26pid%3DApi&f=1)
  
  However there is another problem with sigmoid functions: their gradient has a maximum value of $0.25$! This means that if the weight of a neuron is less than $4$, the error will **diminish** while flowing backward through the net. This becomes progressively worse as we add more layers to the network.
  
  ### How have we solved this issue
  
  Naturally researchers tried to find better weight initialization strategies. However this was hard, because as we saw, small weights are bad and large weights are bad.
  
  One example is [Mishkin et al. 2016](https://arxiv.org/pdf/1511.06422.pdf) who propose a new initialization strategy, but fail to train a deep neural network with sigmoid activations.
  
  Another workaround is to use a different learning rate for each layer ([Xu et al. - Revise Saturated Activation Functions](https://openreview.net/pdf?id=D1VDjyJjXF5jEJ1zfE53))
  
  After a while the Machine Learning community realized that sigmoid functions were ill-suited for deep neural networks and adopted ReLU activations, which have fewer drawbacks and scale better. Nowadays, they have become the de-facto choice for deep learning.
  
  ### Sources
  
  This problem is known for several years and has been well documented (the earliest I could find was in 1994). This was mostly explored in Recurrent Neural Networks.
  
  If you are interested in reading about this, I'd recommend [this post](https://medium.com/@karpathy/yes-you-should-understand-backprop-e2f06eab496b), by Andrej Karpathy.
  
  Some more formal sources on this topic:
  
  * [Hochreiter et al. - Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies](https://www.bioinf.jku.at/publications/older/ch7.pdf)
  * [Hochreiter et al. - Recurrent Neural Net Learning and Vanishing Gradient](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.4.7389&rep=rep1&type=pdf)
  * [Pascanu et al. - On the difficulty of training recurrent neural networks](http://proceedings.mlr.press/v28/pascanu13.pdf)
  * [Bengio et al. - Learning long-term dependencies with gradient descent is difficult](http://ai.dinfo.unifi.it/paolo/ps/tnn-94-gradient.pdf)
  
  
  ---
  
  