Answers.Body: <p>The lower bound in MINE is as follows:</p>
  <p><span class="math-container">$$\widehat{I(X;Z)}_n = \sup_{\theta\in\Theta} \mathbb{E}_{\mathbb{P}_{XZ}^{(n)}}[T_\theta] - \log{\mathbb{E}_{\mathbb{P}_X^{(n)} \otimes \hat{\mathbb{P}}_Z^{(n)}}[e^{T_\theta}]}$$</span></p>
  <p>Here <span class="math-container">$\mathbb{\hat{P}^{(n)}}$</span> denotes the empirical distribution that we get from n i.i.d samples of <span class="math-container">$\mathbb{P}.$</span></p>
  <p>Note that in the above equation, the first term is calculated from the joint distribution while the second term from the marginals of <span class="math-container">$X$</span> and <span class="math-container">$Z$</span>. In the implementation of MINE, these statistics are calculated over the data from a <strong>minibatch</strong>. The marginal distribution is obtained by shuffling the values of Z (or X) along the batch dimension. Hence, in this case, the gradient is as follows.</p>
  <p><span class="math-container">$$\hat{G}_B = \mathbb{E}_B[\nabla_{\theta}T_{\theta}] - \frac{\mathbb{E_B}[\nabla_{\theta}T_{\theta}e^{T_{\theta}}]}{\mathbb{E}_B[e^{T_{\theta}}]},$$</span></p>
  <ol>
  <li>As mentioned, the expectation over the marginals is not calculated
  over the true marginal distribution (i.e. over the entire dataset)
  but from the shuffled samples in the minibatch. Hence, the above gradient <span class="math-container">$G_B$</span> is biased.</li>
  <li>When we maintain an exponential moving average of
  <span class="math-container">$\mathbb{E}_B[e^{T_{\theta}}]$</span>, we kind of incorporate statistics from outside the current minibatch also (i.e. over the entire dataset). This is an attempt to get an approximation of the true marginal estimate. The denominator term in the gradient allows for such a computationally inexpensive bias reduction trick.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T07:20:36.647Z
Answers.LastActivityDate: 2021-03-18T07:20:36.647Z
Answers.UserDisplayName: Mohith Kaameswaran Sakthivel
Body: <p>While reading the Mutual Information Neural Estimation (MINE) paper [<a href="https://arxiv.org/abs/1801.04062" rel="nofollow noreferrer">1</a>] I came across section <strong>3.2 Correcting the bias from the stochastic gradients</strong>. The proposed method requires the computation of the gradient</p>
  <p><span class="math-container">$$\hat{G}_B = \mathbb{E}_B[\nabla_{\theta}T_{\theta}] - \frac{\mathbb{E_B}[\nabla_{\theta}T_{\theta}e^{T_{\theta}}]}{\mathbb{E}_B[e^{T_{\theta}}]},$$</span></p>
  <p>where <span class="math-container">$\mathbb{E}_B$</span> denotes the expectation operation w.r.t. a minibatch <span class="math-container">$B$</span>, and <span class="math-container">$T_{\theta}$</span> is a neural network parameterized by <span class="math-container">$\theta$</span>. The authors claim that this gradient estimation is biased and that can be reduced by simply performing an exponential moving average filtering.</p>
  <p>Can someone give me a hint to understand these two points:</p>
  <ol>
  <li>Why is <span class="math-container">$\hat{G}_B$</span> biased, and</li>
  <li>How does the exponential moving average reduce the bias?</li>
  </ol>
  
Tags: <neural-networks><deep-learning><papers><generative-adversarial-networks><stochastic-gradient-descent>
Title: In the MINE paper, why is $\hat{G}_B$ biased, and how does the exponential moving average reduce the bias?
UserDisplayName: mgbacher
ThreadText: # In the MINE paper, why is $\hat{G}_B$ biased, and how does the exponential moving average reduce the bias?
  
  (Asked by: username_0 on 2020-11-08)
  
  While reading the Mutual Information Neural Estimation (MINE) paper [[1](https://arxiv.org/abs/1801.04062)] I came across section **3.2 Correcting the bias from the stochastic gradients**. The proposed method requires the computation of the gradient
  
  $$\hat{G}\_B = \mathbb{E}\_B[\nabla\_{\theta}T\_{\theta}] - \frac{\mathbb{E\_B}[\nabla\_{\theta}T\_{\theta}e^{T\_{\theta}}]}{\mathbb{E}\_B[e^{T\_{\theta}}]},$$
  
  where $\mathbb{E}\_B$ denotes the expectation operation w.r.t. a minibatch $B$, and $T\_{\theta}$ is a neural network parameterized by $\theta$. The authors claim that this gradient estimation is biased and that can be reduced by simply performing an exponential moving average filtering.
  
  Can someone give me a hint to understand these two points:
  
  1. Why is $\hat{G}\_B$ biased, and
  2. How does the exponential moving average reduce the bias?
  
  **Tags:** neural-networks, deep-learning, papers, generative-adversarial-networks, stochastic-gradient-descent
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-18. Score: 2 | Accepted answer
  
  The lower bound in MINE is as follows:
  
  $$\widehat{I(X;Z)}\_n = \sup\_{\theta\in\Theta} \mathbb{E}\_{\mathbb{P}\_{XZ}^{(n)}}[T\_\theta] - \log{\mathbb{E}\_{\mathbb{P}\_X^{(n)} \otimes \hat{\mathbb{P}}\_Z^{(n)}}[e^{T\_\theta}]}$$
  
  Here $\mathbb{\hat{P}^{(n)}}$ denotes the empirical distribution that we get from n i.i.d samples of $\mathbb{P}.$
  
  Note that in the above equation, the first term is calculated from the joint distribution while the second term from the marginals of $X$ and $Z$. In the implementation of MINE, these statistics are calculated over the data from a **minibatch**. The marginal distribution is obtained by shuffling the values of Z (or X) along the batch dimension. Hence, in this case, the gradient is as follows.
  
  $$\hat{G}\_B = \mathbb{E}\_B[\nabla\_{\theta}T\_{\theta}] - \frac{\mathbb{E\_B}[\nabla\_{\theta}T\_{\theta}e^{T\_{\theta}}]}{\mathbb{E}\_B[e^{T\_{\theta}}]},$$
  
  1. As mentioned, the expectation over the marginals is not calculated
     over the true marginal distribution (i.e. over the entire dataset)
     but from the shuffled samples in the minibatch. Hence, the above gradient $G\_B$ is biased.
  2. When we maintain an exponential moving average of
     $\mathbb{E}\_B[e^{T\_{\theta}}]$, we kind of incorporate statistics from outside the current minibatch also (i.e. over the entire dataset). This is an attempt to get an approximation of the true marginal estimate. The denominator term in the gradient allows for such a computationally inexpensive bias reduction trick.
  
  
  ---
  
  