Answers.Body: <blockquote>
    <p>If n different edges with associated weights <span class="math-container">$w_1, w_2, . . . , w_n$</span> point to this node, then after selecting weights
    with uniform probability from the interval <span class="math-container">$[−α, α]$</span>, the expected total input
    to the node is <span class="math-container">$$\sum_iw_ix_i$$</span>.
    By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution.</p>
  </blockquote>
  
  <p>Note that the input to a node  is <span class="math-container">$$w_1x_1 + w_2x_2 + ... + w_nx_n $$</span></p>
  
  <p>The variance of the total input to a node is:</p>
  
  <p><span class="math-container">$$\sigma^2 = \sum^n_{i=1}E[w_i^2]E[x_i^2] = n(E[w_1^2]E[x_1^2])
  = n([\frac{(\alpha-(-\alpha))^2}{12}][\frac{1}{3}])=n\alpha^2\frac{1}{9}$$</span></p>
  
  <p>since inputs and weights are uncorrelated. </p>
  
  <p>I have taken out the summation and replaced it with <span class="math-container">$n$</span> times the variance of an input and a weight -- this can be done because both are <a href="https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables" rel="nofollow noreferrer">independent and identically distributed</a> in their own right and are independent of each other too. Variance of <span class="math-container">$w_1$</span> is the same as variance of <span class="math-container">$w_n$</span>. </p>
  
  <blockquote>
    <p>Am I missing some implicance of the law of large numbers use for the input to the node?</p>
  </blockquote>
  
  <p>Yes.</p>
  
  <p><em>Implication of Law of Large Numbers:</em></p>
  
  <p>Note that the <em>random variable</em> <span class="math-container">$E(w_ix_i)$</span> in  <span class="math-container">$$E(\sum^n_i(w_ix_i)^2)$$</span> is written as product of the random variables <span class="math-container">$E(w_i)$</span> and <span class="math-container">$E(x_i)$</span> <span class="math-container">$$\sum^n_{i=1}E[w_i^2]E[x_i^2]$$</span>. This can only be written when the two random variables are independent. And the switching of expectation and summation is due to the <a href="https://brilliant.org/wiki/linearity-of-expectation/" rel="nofollow noreferrer">linearity of expectation property</a>.
  How do we get independence here? LLN comes to the rescue.</p>
  
  <p>That comes from the fact that -- according to the law of large numbers -- the <em>random variable</em> (total input) follows a normal distribution which implies the two random variables i.e. the weights and the inputs also follow normal distribution -- and <a href="https://en.wikipedia.org/wiki/Normally_distributed_and_uncorrelated_does_not_imply_independent" rel="nofollow noreferrer">it is also given that the two random variables are uncorrelated and thus they are independent.</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-05-02T10:07:30.893Z
Answers.LastActivityDate: 2019-05-18T07:53:32.840Z
Answers.LastEditDate: 2019-05-18T07:53:32.840Z
Answers.UserDisplayName: naive
Body: <p><a href="https://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf#page=16" rel="nofollow noreferrer">Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1</a> calculates the standard deviation of the output of a hidden neuron.</p>
  
  <p>From:
  <span class="math-container">$$
  \sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2]
  $$</span></p>
  
  <p>When I try what I get is (<a href="https://ai.stackexchange.com/questions/11971/binary-vector-expected-value">with <span class="math-container">$E[x_i^2] = \frac{1}{3}$</span></a> and <span class="math-container">$w_i \in [-\alpha, \alpha]$</span>):</p>
  
  <p><span class="math-container">$$
  \sigma^2 = \sum^n_{i=0}E[w_i^2]E[x_i^2]
  = [n\frac{(\alpha-(-\alpha))^2}{12}][n\frac{1}{3}]=n^2\alpha^2\frac{1}{9}
  $$</span></p>
  
  <p><span class="math-container">$$
  \sigma = \sqrt{n^2\alpha^2\frac{1}{9}}=n\alpha\frac{1}{3}
  $$</span></p>
  
  <p>But Raul Rojas concludes:</p>
  
  <p><span class="math-container">$$
  \sigma = \frac{1}{3}\sqrt{n}\alpha
  $$</span></p>
  
  <p>Am I missing some implicance of the law of large numbers use for the input to the node?</p>
  
  <p>Thank you for your time :)</p>
  
Tags: <neural-networks><math><probability-distribution>
Title: Standard deviation of the total input to a neuron
UserDisplayName: EmmanuelMess
ThreadText: # Standard deviation of the total input to a neuron
  
  (Asked by: username_0 on 2019-04-30)
  
  [Raul Rojas' Neural Networks A Systematic Introduction, section 8.2.1](https://page.mi.fu-berlin.de/rojas/neural/chapter/K8.pdf#page=16) calculates the standard deviation of the output of a hidden neuron.
  
  From:
  $$
  \sigma^2 = \sum^n\_{i=0}E[w\_i^2]E[x\_i^2]
  $$
  
  When I try what I get is ([with $E[x\_i^2] = \frac{1}{3}$](https://ai.stackexchange.com/questions/11971/binary-vector-expected-value) and $w\_i \in [-\alpha, \alpha]$):
  
  $$
  \sigma^2 = \sum^n\_{i=0}E[w\_i^2]E[x\_i^2]
  = [n\frac{(\alpha-(-\alpha))^2}{12}][n\frac{1}{3}]=n^2\alpha^2\frac{1}{9}
  $$
  
  $$
  \sigma = \sqrt{n^2\alpha^2\frac{1}{9}}=n\alpha\frac{1}{3}
  $$
  
  But Raul Rojas concludes:
  
  $$
  \sigma = \frac{1}{3}\sqrt{n}\alpha
  $$
  
  Am I missing some implicance of the law of large numbers use for the input to the node?
  
  Thank you for your time :)
  
  **Tags:** neural-networks, math, probability-distribution
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-05-02. Score: 1 | Accepted answer
  
  > If n different edges with associated weights $w\_1, w\_2, . . . , w\_n$ point to this node, then after selecting weights
  > with uniform probability from the interval $[−α, α]$, the expected total input
  > to the node is $$\sum\_iw\_ix\_i$$.
  > By the law of large numbers we can also assume that the total input to the node has a Gaussian distribution.
  
  Note that the input to a node is $$w\_1x\_1 + w\_2x\_2 + ... + w\_nx\_n $$
  
  The variance of the total input to a node is:
  
  $$\sigma^2 = \sum^n\_{i=1}E[w\_i^2]E[x\_i^2] = n(E[w\_1^2]E[x\_1^2])
  = n([\frac{(\alpha-(-\alpha))^2}{12}][\frac{1}{3}])=n\alpha^2\frac{1}{9}$$
  
  since inputs and weights are uncorrelated.
  
  I have taken out the summation and replaced it with $n$ times the variance of an input and a weight -- this can be done because both are [independent and identically distributed](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables) in their own right and are independent of each other too. Variance of $w\_1$ is the same as variance of $w\_n$.
  
  > Am I missing some implicance of the law of large numbers use for the input to the node?
  
  Yes.
  
  *Implication of Law of Large Numbers:*
  
  Note that the *random variable* $E(w\_ix\_i)$ in $$E(\sum^n\_i(w\_ix\_i)^2)$$ is written as product of the random variables $E(w\_i)$ and $E(x\_i)$ $$\sum^n\_{i=1}E[w\_i^2]E[x\_i^2]$$. This can only be written when the two random variables are independent. And the switching of expectation and summation is due to the [linearity of expectation property](https://brilliant.org/wiki/linearity-of-expectation/).
  How do we get independence here? LLN comes to the rescue.
  
  That comes from the fact that -- according to the law of large numbers -- the *random variable* (total input) follows a normal distribution which implies the two random variables i.e. the weights and the inputs also follow normal distribution -- and [it is also given that the two random variables are uncorrelated and thus they are independent.](https://en.wikipedia.org/wiki/Normally_distributed_and_uncorrelated_does_not_imply_independent)
  
  
  ---
  
  