Answers.Body: <p>The most effective way to prevent both the forward and backward propagation of <em>exploding</em> is keeping the weights in a small range. The main way this is accomplished is through their <strong>initialization</strong>.</p>
  
  <p>For example in the case of <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="nofollow noreferrer">He initialization</a>, the authors show (given some assumptions) that the variance of the output of the final layer <span class="math-container">$L$</span> of the network is:</p>
  
  <p><span class="math-container">$$
  Var[y_L] = Var[y_1] \left( \prod_{i=2}^L{\frac{1}{2} \, n_l \, Var[w_l]} \right)
  $$</span></p>
  
  <p>where <span class="math-container">$n_l$</span> and <span class="math-container">$w_l$</span> are the number of connections and weights of layer <span class="math-container">$l$</span>. In order to keep the outputs from exploding the product above should not exponentially magnify its inputs. In order to do this the authors elect to initialize the weights so that:</p>
  
  <p><span class="math-container">$$
  \frac{1}{2} \, n_l \, Var[w_l] = 1
  $$</span></p>
  
  <p>Now this is helps keeping the outputs from exploding. They then go and prove that the same strategy helps preventing the gradients from exploding. </p>
  
  <p>Another similar strategy is the so-called <a href="http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">Glorot (or Xavier) Initialization</a>.
  These techniques are <strong>extremely effective</strong> in helping the models converge!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-29T15:42:01.523Z
Answers.Text: I had thought maybe this could be a solution, but assumed there was a way of applying this such that it was independent of the number of weights/layers. Thankyou for linking the paper, I'll give it a read when I get the time
Answers.UserDisplayName: Recessive
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-28T09:10:08.543Z
Answers.LastActivityDate: 2019-06-28T09:10:08.543Z
Answers.UserDisplayName: Djib2011
Body: <p>Take AlexNet for example:</p>
  
  <p><a href="https://i.sstatic.net/YZce8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/YZce8.png" alt="combined gpu stream of alexnet"></a></p>
  
  <p>In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:</p>
  
  <p>Say I have a weight matrix of <code>[-1,-2,3,4]</code> and inputs of <code>[ReLU(4), ReLU(5), ReLU(-2), Relu(-3)]</code>. The resultant matrix from these will have large numbers for the inputs of <code>ReLU(4)</code> and <code>ReLU(5)</code>, and 0 for <code>ReLU(-2)</code> and <code>ReLU(-3)</code>. If there are even just a few more layers, the numbers are quick to either explode or be 0.</p>
  
  <p>How is this typically combated? How do you keep these numbers towards 0? I understand you can take subtract the mean at the end of each layer, but for a layer that is already in the millions, subtracting the mean will still result in thousands.</p>
  
Tags: <machine-learning><convolutional-neural-networks><relu>
Title: How are exploding numbers in a forward pass of a CNN combated?
UserDisplayName: Recessive
ThreadText: # How are exploding numbers in a forward pass of a CNN combated?
  
  (Asked by: username_0 on 2019-06-28)
  
  Take AlexNet for example:
  
  [![combined gpu stream of alexnet](https://i.sstatic.net/YZce8.png)](https://i.sstatic.net/YZce8.png)
  
  In this case, only the activation function ReLU is used. Due to the fact ReLU cannot be saturated, it instead explodes, like in the following example:
  
  Say I have a weight matrix of `[-1,-2,3,4]` and inputs of `[ReLU(4), ReLU(5), ReLU(-2), Relu(-3)]`. The resultant matrix from these will have large numbers for the inputs of `ReLU(4)` and `ReLU(5)`, and 0 for `ReLU(-2)` and `ReLU(-3)`. If there are even just a few more layers, the numbers are quick to either explode or be 0.
  
  How is this typically combated? How do you keep these numbers towards 0? I understand you can take subtract the mean at the end of each layer, but for a layer that is already in the millions, subtracting the mean will still result in thousands.
  
  **Tags:** machine-learning, convolutional-neural-networks, relu
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-06-28. Score: 2 | Accepted answer
  
  The most effective way to prevent both the forward and backward propagation of *exploding* is keeping the weights in a small range. The main way this is accomplished is through their **initialization**.
  
  For example in the case of [He initialization](https://arxiv.org/pdf/1502.01852.pdf), the authors show (given some assumptions) that the variance of the output of the final layer $L$ of the network is:
  
  $$
  Var[y\_L] = Var[y\_1] \left( \prod\_{i=2}^L{\frac{1}{2} \, n\_l \, Var[w\_l]} \right)
  $$
  
  where $n\_l$ and $w\_l$ are the number of connections and weights of layer $l$. In order to keep the outputs from exploding the product above should not exponentially magnify its inputs. In order to do this the authors elect to initialize the weights so that:
  
  $$
  \frac{1}{2} \, n\_l \, Var[w\_l] = 1
  $$
  
  Now this is helps keeping the outputs from exploding. They then go and prove that the same strategy helps preventing the gradients from exploding.
  
  Another similar strategy is the so-called [Glorot (or Xavier) Initialization](http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf).
  These techniques are **extremely effective** in helping the models converge!
  
  > **Comments:**
  > **username_0**: I had thought maybe this could be a solution, but assumed there was a way of applying this such that it was independent of the number of weights/layers. Thankyou for linking the paper, I'll give it a read when I get the time  
  
  ---
  
  