Body: <p>The title is one of the special things in <a href="https://arxiv.org/pdf/1710.10196.pdf" rel="nofollow noreferrer">Progressive GAN</a>, a paper of the NVIDIA team. By using this method, they introduced that</p>
  <blockquote>
  <p>Our approach ensures that the dynamic range, and thus the learning speed, is the same for all weights.</p>
  </blockquote>
  <p>In details, they inited all learnable parameters by normal distribution <span class="math-container">$N(0,1)$</span>. During training time, each forward time, they will scale the result with <strong>per-layer normalization constant from He's initializer</strong></p>
  <p>I reproduced the code from <a href="https://github.com/facebookresearch/pytorch_GAN_zoo" rel="nofollow noreferrer">pytorch GAN zoo Github's repo</a></p>
  <pre><code>def forward(self, x, equalized):
      # generate He constant depend on the size of tensor W
      size = self.module.weight.size()
      fan_in = prod(size[1:])
      weight = math.sqrt(2.0 / fan_in)
      '''
      A module example:
  
      import torch.nn as nn
      module = nn.Conv2d(nChannelsPrevious, nChannels, kernelSize, padding=padding, bias=bias) 
      '''
      x = self.module(x)
  
      if equalized:
          x *= self.weight
      return x
  </code></pre>
  <p>At first, I thought the <strong>He constant</strong> will be <span class="math-container">$c = \frac{\sqrt{2}}{\sqrt{n_l}}$</span> as <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="nofollow noreferrer">He's paper</a>. Normally, <span class="math-container">$n_l &gt; 2$</span> so <span class="math-container">$w_l$</span> can be scale up which lead to the gradient in backpropagation is increase as the formula in ProGan's paper <span class="math-container">$\hat{w}_i=\frac{w_i}{c}$</span> <span class="math-container">$\rightarrow$</span> prevent vanishing gradient.</p>
  <p>However, the code shows that <span class="math-container">$\hat{w}_i=w_i*c$</span>.</p>
  <p>In summary, I can't understand why to scale down the parameter many times during training will help the learning speed be more stable.</p>
  <p>I asked this question on some communities e.g: StackOverflow, mathematics, Data Science, and still haven't had an answer.</p>
  <p>Please help me explain it, thank you!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-29T16:24:22.990Z
Comments.Text: What do you mean by "He constant"? And which parameter are you referring to when you say "I can't understand why to scale down the parameter many times during"? Moreover, note that I changed the title of your post to be more descriptive. Make sure that's your question in the title.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-09-30T03:59:30.853Z
Comments.Text: Thank you for fixing my title, I've not had the experience to ask a correct question. I said "He constant" because the paper mentioned it. At first, I didn't understand so I found the code and got the equation above. About my question, the parameters I mentioned here is all learnable parameters that init by Normal distribution, if you want I can quote a part of the paper into my question.
Comments.UserDisplayName: CuCaRot
Tags: <training><optimization><gradient-descent><papers><generative-adversarial-networks>
Title: Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?
UserDisplayName: CuCaRot
ThreadText: # Why scaling down the parameter many times during training will help the learning speed be the same for all weights in Progressive GAN?
  
  (Asked by: username_0 on 2020-09-29)
  
  The title is one of the special things in [Progressive GAN](https://arxiv.org/pdf/1710.10196.pdf), a paper of the NVIDIA team. By using this method, they introduced that
  
  > Our approach ensures that the dynamic range, and thus the learning speed, is the same for all weights.
  
  In details, they inited all learnable parameters by normal distribution $N(0,1)$. During training time, each forward time, they will scale the result with **per-layer normalization constant from He's initializer**
  
  I reproduced the code from [pytorch GAN zoo Github's repo](https://github.com/facebookresearch/pytorch_GAN_zoo)
  
  ```
  def forward(self, x, equalized):
      # generate He constant depend on the size of tensor W
      size = self.module.weight.size()
      fan_in = prod(size[1:])
      weight = math.sqrt(2.0 / fan_in)
      '''
      A module example:
  
      import torch.nn as nn
      module = nn.Conv2d(nChannelsPrevious, nChannels, kernelSize, padding=padding, bias=bias) 
      '''
      x = self.module(x)
  
      if equalized:
          x *= self.weight
      return x
  
  ```
  
  At first, I thought the **He constant** will be $c = \frac{\sqrt{2}}{\sqrt{n\_l}}$ as [He's paper](https://arxiv.org/pdf/1502.01852.pdf). Normally, $n\_l > 2$ so $w\_l$ can be scale up which lead to the gradient in backpropagation is increase as the formula in ProGan's paper $\hat{w}\_i=\frac{w\_i}{c}$ $\rightarrow$ prevent vanishing gradient.
  
  However, the code shows that $\hat{w}\_i=w\_i\*c$.
  
  In summary, I can't understand why to scale down the parameter many times during training will help the learning speed be more stable.
  
  I asked this question on some communities e.g: StackOverflow, mathematics, Data Science, and still haven't had an answer.
  
  Please help me explain it, thank you!
  
  **Tags:** training, optimization, gradient-descent, papers, generative-adversarial-networks
  
  > **Comments:**
  > **username_1**: What do you mean by "He constant"? And which parameter are you referring to when you say "I can't understand why to scale down the parameter many times during"? Moreover, note that I changed the title of your post to be more descriptive. Make sure that's your question in the title.  
  > **username_0**: Thank you for fixing my title, I've not had the experience to ask a correct question. I said "He constant" because the paper mentioned it. At first, I didn't understand so I found the code and got the equation above. About my question, the parameters I mentioned here is all learnable parameters that init by Normal distribution, if you want I can quote a part of the paper into my question.  
  
  ---
  
  ## 0 answers
  
  