Body: <p>How does:<br />
  <span class="math-container">$$\text{Var}(y) \approx \sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$$</span>
  approximate variance?</p>
  <p>I'm currently reading <em><a href="https://arxiv.org/abs/1703.04977" rel="nofollow noreferrer">What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision</a></em>, and the authors wrote the above formula for the approximate estimation for the variance. I'm confused how the above is an approximation for <span class="math-container">$\frac{\sum(y-\bar{y})^2}{N-1}$</span>. So, in the above equation, they're using a Bayesian Neural Network to quantify uncertainty. <span class="math-container">$\sigma$</span> is the predictive variance (kind of confused how they get this). <span class="math-container">$x$</span> is the input and <span class="math-container">$y$</span> is the label for the classification. <span class="math-container">$f^{\hat{W_t}}(\cdot)$</span> output a mean to a Gaussian distribution, with <span class="math-container">$\sigma$</span> being the SD for that distribution and <span class="math-container">$T$</span> is a predefined number of samples because the gradient is evaluated using Monte Carlo sampling.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T05:15:37.493Z
Comments.Text: @DuttaA sadly they don't give dimensions. I think the reason being you can adjust the model size to fit whatever you need, so there's no strict requirement in terms of dimensionality. If you like you can just think of them as vectors I think that's reasonable.
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T05:19:08.197Z
Comments.Text: dot product maybe?
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T05:24:00.360Z
Comments.Text: it's actually a little confusing I'm looking at the paper right now. They denote $T$ as MC samples, but then they also use the same style of $T$ over $f()$ and $E(Y)$, I assume it's transpose, but maybe they actually are putting real values to the power of $T$? That doesn't really make sense though.
Comments.UserDisplayName: user8714896
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T06:50:23.113Z
Comments.Text: @user8714896 You can try contact the authors.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T09:56:08.230Z
Comments.Text: If this is from equation (4), you've got a typo and the transpose in the sum term should be on the other element (so it is an inner product, and the dimensions are at least plausible).
Comments.UserDisplayName: htl
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T10:30:49.107Z
Comments.Text: @htl You can and should propose an edit to the post if you think there are typos there.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-20T22:51:02.493Z
Comments.Text: Why did you delete this post? I was trying to write an answer, although that answer could not be satisfactory. In any case, I don't know when I will finish writing it.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-21T00:03:52.760Z
Comments.Text: @nbro go ahead and put an answer. I posted on stats stack exchange to see if stats people had any intuition about this, that's why I deleted it, but I'm eagerly waiting to see what you write up.
Comments.UserDisplayName: user8714896
Tags: <bayesian-deep-learning><bayesian-neural-networks><uncertainty-quantification><variance>
Title: Why does this formula $\sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$ approximate the variance?
UserDisplayName: user8714896
ThreadText: # Why does this formula $\sigma^2 + \frac{1}{T}\sum_{t=1}^Tf^{\hat{W_t}}(x)^Tf^{\hat{W_t}}(x_t)-E(y)^TE(y)$ approximate the variance?
  
  (Asked by: username_0 on 2021-02-20)
  
  How does:  
  $$\text{Var}(y) \approx \sigma^2 + \frac{1}{T}\sum\_{t=1}^Tf^{\hat{W\_t}}(x)^Tf^{\hat{W\_t}}(x\_t)-E(y)^TE(y)$$
  approximate variance?
  
  I'm currently reading *[What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision](https://arxiv.org/abs/1703.04977)*, and the authors wrote the above formula for the approximate estimation for the variance. I'm confused how the above is an approximation for $\frac{\sum(y-\bar{y})^2}{N-1}$. So, in the above equation, they're using a Bayesian Neural Network to quantify uncertainty. $\sigma$ is the predictive variance (kind of confused how they get this). $x$ is the input and $y$ is the label for the classification. $f^{\hat{W\_t}}(\cdot)$ output a mean to a Gaussian distribution, with $\sigma$ being the SD for that distribution and $T$ is a predefined number of samples because the gradient is evaluated using Monte Carlo sampling.
  
  **Tags:** bayesian-deep-learning, bayesian-neural-networks, uncertainty-quantification, variance
  
  > **Comments:**
  > **username_0**: @DuttaA sadly they don't give dimensions. I think the reason being you can adjust the model size to fit whatever you need, so there's no strict requirement in terms of dimensionality. If you like you can just think of them as vectors I think that's reasonable.  
  > **username_0**: dot product maybe?  
  > **username_0**: it's actually a little confusing I'm looking at the paper right now. They denote $T$ as MC samples, but then they also use the same style of $T$ over $f()$ and $E(Y)$, I assume it's transpose, but maybe they actually are putting real values to the power of $T$? That doesn't really make sense though.  
  > **username_1**: @username_0 You can try contact the authors.  
  > **username_2**: If this is from equation (4), you've got a typo and the transpose in the sum term should be on the other element (so it is an inner product, and the dimensions are at least plausible).  
  > **username_3**: @username_2 You can and should propose an edit to the post if you think there are typos there.  
  > **username_3**: Why did you delete this post? I was trying to write an answer, although that answer could not be satisfactory. In any case, I don't know when I will finish writing it.  
  > **username_0**: @username_3 go ahead and put an answer. I posted on stats stack exchange to see if stats people had any intuition about this, that's why I deleted it, but I'm eagerly waiting to see what you write up.  
  
  ---
  
  ## 0 answers
  
  