Answers.Body: <p>The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called <em>Xavier</em> or <em>Glorot</em> (even though it was named by the authors <em>normalized activation</em> in the original <a href="https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf" rel="nofollow noreferrer">paper</a>). The Xavier activation is also simply an activation function and not a new kind of probability distribution.</p>
  <p>To answer the question, both functions simply sample values from either a uniform or normal distribution. What change is how the parameters that define the distributions are estimated.</p>
  <p>Focusing on the Kaiming activation, since that's the question asked, the formulas used to determine the correct parameters of the uniform and normal distributions to sample values from are:</p>
  <ul>
  <li><p>Uniform: <span class="math-container">$$\boldsymbol{\mathit{U}}(-\sqrt{6 / n_{j}}, +\sqrt{6 / n_{j}})   $$</span></p>
  </li>
  <li><p>Normal:
  <span class="math-container">$$\boldsymbol{\mathbf{\mathit{N}}}(0, \sqrt{2/n_{j}})$$</span></p>
  </li>
  </ul>
  <p>nj is the generic notation used in the original paper, in other sources like the <a href="https://pytorch.org/docs/stable/nn.init.html" rel="nofollow noreferrer">pytorch documentation</a> the notation used is <em>fan_in</em> or <em>fan_out</em>. Without explaining the details, fan_in means number of hidden units of the input layers for the weight matrix being initialized (or number of rows of the weight matrix to initialize), while fan_out means number of hidden units of the output layers for the weight matrix being initialized (or number of columns of the weight matrix to initialize).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-11-01T11:52:19.063Z
Answers.LastActivityDate: 2021-11-01T11:52:19.063Z
Answers.UserDisplayName: Edoardo Guerriero
Body: <p>A probability density function is a real-valued function that roughly gives the density of probability at a particular value of a random variable.</p>
  <p>For example, the probability density function of a normal random variable is given by</p>
  <p><span class="math-container">$$f(x) = \dfrac{1}{2\sigma \sqrt{2\pi}} e^{-{\LARGE(}\dfrac{x-\mu}{\sigma}{\LARGE)}^2}$$</span></p>
  <p>Uniform Kaiming He probability distribution function is used for initialization of weights in <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py#L41" rel="nofollow noreferrer">Convolutional neural networks in PyTorch</a> and the distribution function was initially mentioned in the research paper titled <a href="https://arxiv.org/pdf/1502.01852.pdf" rel="nofollow noreferrer">Delving Deep into Rectifiers:
  Surpassing Human-Level Performance on ImageNet Classification</a> by <em>Kaiming He et al.</em> I think.</p>
  <p>What is the analytical formula for the Kaiming He probability density function?</p>
  
Tags: <convolutional-neural-networks><math><probability-distribution><weights-initialization>
Title: What is the analytical formula for "Kaiming He" probability density function?
UserDisplayName: hanugm
ThreadText: # What is the analytical formula for "Kaiming He" probability density function?
  
  (Asked by: username_0 on 2021-11-01)
  
  A probability density function is a real-valued function that roughly gives the density of probability at a particular value of a random variable.
  
  For example, the probability density function of a normal random variable is given by
  
  $$f(x) = \dfrac{1}{2\sigma \sqrt{2\pi}} e^{-{\LARGE(}\dfrac{x-\mu}{\sigma}{\LARGE)}^2}$$
  
  Uniform Kaiming He probability distribution function is used for initialization of weights in [Convolutional neural networks in PyTorch](https://github.com/pytorch/pytorch/blob/master/torch/nn/modules/conv.py#L41) and the distribution function was initially mentioned in the research paper titled [Delving Deep into Rectifiers:
  Surpassing Human-Level Performance on ImageNet Classification](https://arxiv.org/pdf/1502.01852.pdf) by *Kaiming He et al.* I think.
  
  What is the analytical formula for the Kaiming He probability density function?
  
  **Tags:** convolutional-neural-networks, math, probability-distribution, weights-initialization
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-11-01. Score: 1
  
  The activation function proposed by He et al. is not a new probability function of its own kind. It's an improvement over a previously proposed activation function now called *Xavier* or *Glorot* (even though it was named by the authors *normalized activation* in the original [paper](https://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf)). The Xavier activation is also simply an activation function and not a new kind of probability distribution.
  
  To answer the question, both functions simply sample values from either a uniform or normal distribution. What change is how the parameters that define the distributions are estimated.
  
  Focusing on the Kaiming activation, since that's the question asked, the formulas used to determine the correct parameters of the uniform and normal distributions to sample values from are:
  
  * Uniform: $$\boldsymbol{\mathit{U}}(-\sqrt{6 / n\_{j}}, +\sqrt{6 / n\_{j}}) $$
  * Normal:
    $$\boldsymbol{\mathbf{\mathit{N}}}(0, \sqrt{2/n\_{j}})$$
  
  nj is the generic notation used in the original paper, in other sources like the [pytorch documentation](https://pytorch.org/docs/stable/nn.init.html) the notation used is *fan\_in* or *fan\_out*. Without explaining the details, fan\_in means number of hidden units of the input layers for the weight matrix being initialized (or number of rows of the weight matrix to initialize), while fan\_out means number of hidden units of the output layers for the weight matrix being initialized (or number of columns of the weight matrix to initialize).
  
  
  ---
  
  