Answers.Body: <p>Weights initialisation is strictly related to the vanishing/exploding gradient problem. For a complete explanation, please check this awesome <a href="https://www.deeplearning.ai/ai-notes/initialization/" rel="nofollow noreferrer">page</a> (also from deeplearning.ai). Here I'll summarise the main concepts:</p>
  
  <ul>
  <li>initialising weight all to zero will cause all weights to have the same derivative value with respect to the loss function, hence the network would be incapable of learning anything. </li>
  <li><p>initialising to zero the biases has no drawback since they are constants (no effect at all when computing the derivative anyway).</p></li>
  <li><p>initialising weights with too high or too small values will lead to an exploding gradient (oscillating gradient values without convergence) or a vanishing gradient (small gradient values variation that converge before reaching the loss global minimum).     </p></li>
  </ul>
  
  <p>In order to avoid these problems, a method called <em>Xavier initialisation</em> has been proposed: the weights should be initialised in such a way that they will generate activations scores with a distribution that has:</p>
  
  <ul>
  <li>Mean 0</li>
  <li>Constant variance across layers (i.e. no vanishing/exploding) </li>
  </ul>
  
  <p>The value "np.sqrt(layer_dims[l - 1])" pup up when imposing the second constrain.  For a formal prove check the page I linked. To grasp the concept just focus on the fact that the variance of the weights of a layer depends on the amount of nodes of the previous layer. This mean that for layers preceded by layers with a big amount of hidden nodes, the weights will be initialised with a small variance, and this is ok cause we don't want a small bunch nodes to have stronger influence on the subsequent activations. But in layers which instead are preceded by layers with a small amount of nodes, it's ok to allow the weights to vary more. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T03:34:20.783Z
Answers.LastActivityDate: 2020-04-14T03:34:20.783Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>I did watch the course DeepLearning of Andrew Ng and he told that we should create parameter <code>w</code> small like:</p>
  
  <pre><code>parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) ** 0.001 
  </code></pre>
  
  <p>But in the last application assignment. They choose another way:</p>
  
  <pre><code>layers_dims = [12288, 20, 7, 5, 1]
  def initialize_parameters_deep(layer_dims):
      np.random.seed(3)
      parameters = {}
      L = len(layer_dims)
  
      for l in range(1, L):
          parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) / np.sqrt(layer_dims[l - 1])
          parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))
          assert (parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l - 1]))
          assert (parameters['b' + str(l)].shape == (layer_dims[l], 1))
      return parameters
  </code></pre>
  
  <p>And the result of this way is very good but if I choose <code>w</code> like the old above, It's just have <code>34%</code> correct!</p>
  
  <p>So do you can explain ?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-14T13:19:23.543Z
Comments.Text: Hi and welcome to AI SE! You accepted the answer https://ai.stackexchange.com/a/20242/2444, but if you feel that the answer is also valuable, you should also upvote it by clicking on the arrow that points upwards close to the answer. In this way, you encourage the user to provide more answers!
Comments.UserDisplayName: nbro
Tags: <deep-learning>
Title: How to choosing the random value for parameter w in deep learning network?
UserDisplayName: nguyen son
ThreadText: # How to choosing the random value for parameter w in deep learning network?
  
  (Asked by: username_0 on 2020-04-12)
  
  I did watch the course DeepLearning of Andrew Ng and he told that we should create parameter `w` small like:
  
  ```
  parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) ** 0.001 
  
  ```
  
  But in the last application assignment. They choose another way:
  
  ```
  layers_dims = [12288, 20, 7, 5, 1]
  def initialize_parameters_deep(layer_dims):
      np.random.seed(3)
      parameters = {}
      L = len(layer_dims)
  
      for l in range(1, L):
          parameters['W' + str(l)] = np.random.randn(layer_dims[l], layer_dims[l - 1]) / np.sqrt(layer_dims[l - 1])
          parameters['b' + str(l)] = np.zeros((layer_dims[l], 1))
          assert (parameters['W' + str(l)].shape == (layer_dims[l], layer_dims[l - 1]))
          assert (parameters['b' + str(l)].shape == (layer_dims[l], 1))
      return parameters
  
  ```
  
  And the result of this way is very good but if I choose `w` like the old above, It's just have `34%` correct!
  
  So do you can explain ?
  
  **Tags:** deep-learning
  
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! You accepted the answer https://ai.stackexchange.com/a/20242/2444, but if you feel that the answer is also valuable, you should also upvote it by clicking on the arrow that points upwards close to the answer. In this way, you encourage the user to provide more answers!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-14. Score: 3 | Accepted answer
  
  Weights initialisation is strictly related to the vanishing/exploding gradient problem. For a complete explanation, please check this awesome [page](https://www.deeplearning.ai/ai-notes/initialization/) (also from deeplearning.ai). Here I'll summarise the main concepts:
  
  * initialising weight all to zero will cause all weights to have the same derivative value with respect to the loss function, hence the network would be incapable of learning anything.
  * initialising to zero the biases has no drawback since they are constants (no effect at all when computing the derivative anyway).
  * initialising weights with too high or too small values will lead to an exploding gradient (oscillating gradient values without convergence) or a vanishing gradient (small gradient values variation that converge before reaching the loss global minimum).
  
  In order to avoid these problems, a method called *Xavier initialisation* has been proposed: the weights should be initialised in such a way that they will generate activations scores with a distribution that has:
  
  * Mean 0
  * Constant variance across layers (i.e. no vanishing/exploding)
  
  The value "np.sqrt(layer\_dims[l - 1])" pup up when imposing the second constrain. For a formal prove check the page I linked. To grasp the concept just focus on the fact that the variance of the weights of a layer depends on the amount of nodes of the previous layer. This mean that for layers preceded by layers with a big amount of hidden nodes, the weights will be initialised with a small variance, and this is ok cause we don't want a small bunch nodes to have stronger influence on the subsequent activations. But in layers which instead are preceded by layers with a small amount of nodes, it's ok to allow the weights to vary more.
  
  
  ---
  
  