Answers.Body: <p>The KL divergence is just <em>not symmetric</em>, and so changing <span class="math-container">$q$</span> for <span class="math-container">$p$</span>, and vice-versa, gives you a different behavior because the expectation is computed on a different distribution.</p>
  <ul>
  <li>In the first plot, you see the <strong>forward KL</strong>, <span class="math-container">$KL(q\ \|\ p)$</span>, that encourages the approximating distribution <span class="math-container">$q(z)$</span> (the one you topically learn) to cover the mode of <span class="math-container">$p(z)$</span>: this is called <em>mode-seeking</em> behavior, which tries to match the most-likely point of the distribution <span class="math-container">$p$</span>.</li>
  <li>The other plot shows, instead, the <strong>reverse KL</strong>, <span class="math-container">$KL(p\ \|\ q)$</span>, that tries to match the support of <span class="math-container">$p(z)$</span>, however loosing the mode in the approximating distribution.</li>
  </ul>
  <p>To recap, the forward KL tries to put probability mass in the area of the distribution where the probability of samples is higher (in this example, the area where the std is 1), while the reverse KL tries just to cover all the distribution's support placing some non-zero probability everywhere: that's why the second, bigger, red circle covers the most of the green contours.</p>
  <p><strong>Update:</strong></p>
  <blockquote>
  <p>Which is the same variables?</p>
  </blockquote>
  <p>The (random) variables <span class="math-container">$z_1$</span> and <span class="math-container">$z_2$</span> define the &quot;space&quot; of values that you can observe, e.g. you're given a dataset of samples taken from <span class="math-container">$p(z)$</span>, i.e. the green distribution, where <span class="math-container">$z \in \mathbb [0,1]^2$</span>. Basically, the approximating distribution, <span class="math-container">$q(z)$</span>, is defined on the <em>same</em> space where the variables <span class="math-container">$z_1$</span> and <span class="math-container">$z_2$</span> lie.</p>
  <blockquote>
  <p>Which is the two distributions?</p>
  </blockquote>
  <p>I think Bishop assumes <span class="math-container">$q(z)$</span> to be defined as a <em>mean field</em> approximation, which should correspond to <span class="math-container">$q(z) = \prod_{i\in\{1,2\}} q(z_i; \mu_i, \sigma)$</span> where <span class="math-container">$\mu_i$</span> and <span class="math-container">$\sigma$</span> are the parameters you want to learn to approximate <span class="math-container">$p(z)$</span>. Basically, you consider two tractable distributions, e.g. a Gaussian, and learn the parameters of each &quot;simple&quot; distribution. Then, to get the final, more complex, distribution you perform the mean field assumption which is about doing a product among each <span class="math-container">$q_i(z_i)$</span>. Also I put <span class="math-container">$\sigma$</span> and not <span class="math-container">$\sigma_i$</span> since from the figures the variance seems to be independent from <span class="math-container">$z_i$</span>. For example, the mean field assumption does not enable you to learn the correlation between each <span class="math-container">$z_i$</span>, so this may explain the same variance across dimensions.</p>
  <blockquote>
  <p>What is the product? Does it mean the probability density function?</p>
  </blockquote>
  <p>I think a partial answer is above. Anyway, is a product over the pdf of each <span class="math-container">$p_i$</span> and <span class="math-container">$q_i$</span> to yield, respectively, <span class="math-container">$p(z)$</span> and <span class="math-container">$q(z)$</span>. But in the case of <span class="math-container">$p(z)$</span> you also account for the
  correlations when doing the product, instead in <span class="math-container">$q(z)$</span> independence is assumed.</p>
  <p>In the end, you can define <span class="math-container">$p(z) = \mathcal N(\mu,\Sigma)$</span> where <span class="math-container">$\mu=[0.5,0.5]$</span> and <span class="math-container">$\Sigma=\begin{bmatrix} \sigma_{11} &amp; \sigma_{12} \\ \sigma_{21} &amp; \sigma_{22} \end{bmatrix}$</span>, where each <span class="math-container">$\sigma_{ij}$</span> (for <span class="math-container">$i=1,2$</span> and <span class="math-container">$j=1,2$</span>) tells you the correlation between variable <span class="math-container">$z_i$</span> and <span class="math-container">$z_j$</span> that has the effect of shrinking or expanding and rotating (more importantly) the green Gaussian. Instead, <span class="math-container">$q(z)=\prod_i \mathcal N(\hat\mu_i,\hat\sigma_i)$</span>, since is an approximation, has no such capabilities (e.g., rotation) and so you get a perfect circle, smaller or larger according to the KL you minimize to learn the parameters: <span class="math-container">$\hat\mu_1 = \hat\mu_2 = 0.5$</span> and <span class="math-container">$\hat\sigma_1 = \hat\sigma_2 = \hat \sigma$</span> or <span class="math-container">$\hat\Sigma=\begin{bmatrix} \hat\sigma &amp; 0 \\ 0 &amp; \hat\sigma \end{bmatrix}$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-21T19:22:04.610Z
Answers.Text: @diffusionstable I'm glad the answer helped you. I've edited it trying to answer the other questions too. Have a look and let me know!
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-22T14:30:25.313Z
Answers.Text: Thank you. It's a bit complicated, so I'll try to read it slowly!
Answers.UserDisplayName: diffusion stable
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-20T13:41:03.553Z
Answers.LastActivityDate: 2023-07-21T19:20:59.253Z
Answers.LastEditDate: 2023-07-21T19:20:59.253Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>On page 468 of '<a href="https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf" rel="nofollow noreferrer">Pattern Recognition and Machine Learning</a>', what does 'the <strong>same variables</strong> given by the <strong>product</strong> of <strong>two</strong> independent univariate Gaussian <strong>distributions</strong>' mean?</p>
  <p>The PDF says,</p>
  <blockquote>
  <p>The green contours corresponding to 1, 2, and 3 standard deviations
  for a correlated Gaussian distribution p(z) over two variables z1 and
  z2, and the red contours represent the corresponding levels for an
  approximating distribution q(z) over the same variables given by the
  product of two independent univariate Gaussian distributions whose
  parameters are obtained by minimization of (a) the KullbackLeibler
  divergence KL(q||p), and (b) the reverse Kullback-Leibler divergence
  KL(p||q).</p>
  </blockquote>
  <p>but I don't understand the explanation clearly.</p>
  <p>Which is the same variables?<br>
  Which is the two distributions?<br>
  What is the product? Does it mean the probability density function?<br>
  Thanks.</p>
  <p><a href="https://i.sstatic.net/zSLBv.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/zSLBv.png" alt="enter image description here" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T13:04:01.287Z
Comments.Text: Can you please ask a **specific question**? "I don't understand the explanation clearly." is not a question and does make clear what you don't understand. Please, once you've decide what your specific question is, put it also in the title. Thanks.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-07-20T14:28:59.393Z
Comments.Text: I added specific questions  such as 'same variables' and 'two distributions'. Thanks.
Comments.UserDisplayName: diffusion stable
Tags: <machine-learning><deep-learning><generative-adversarial-networks><variational-autoencoder>
Title: Comparison of the two alternative forms for the KL divergence
UserDisplayName: diffusion stable
ThreadText: # Comparison of the two alternative forms for the KL divergence
  
  (Asked by: username_0 on 2023-07-20)
  
  On page 468 of '[Pattern Recognition and Machine Learning](https://www.microsoft.com/en-us/research/uploads/prod/2006/01/Bishop-Pattern-Recognition-and-Machine-Learning-2006.pdf)', what does 'the **same variables** given by the **product** of **two** independent univariate Gaussian **distributions**' mean?
  
  The PDF says,
  
  > The green contours corresponding to 1, 2, and 3 standard deviations
  > for a correlated Gaussian distribution p(z) over two variables z1 and
  > z2, and the red contours represent the corresponding levels for an
  > approximating distribution q(z) over the same variables given by the
  > product of two independent univariate Gaussian distributions whose
  > parameters are obtained by minimization of (a) the KullbackLeibler
  > divergence KL(q||p), and (b) the reverse Kullback-Leibler divergence
  > KL(p||q).
  
  but I don't understand the explanation clearly.
  
  Which is the same variables?  
  Which is the two distributions?  
  What is the product? Does it mean the probability density function?  
  Thanks.
  
  [![enter image description here](https://i.sstatic.net/zSLBv.png)](https://i.sstatic.net/zSLBv.png)
  
  **Tags:** machine-learning, deep-learning, generative-adversarial-networks, variational-autoencoder
  
  > **Comments:**
  > **username_1**: Can you please ask a **specific question**? "I don't understand the explanation clearly." is not a question and does make clear what you don't understand. Please, once you've decide what your specific question is, put it also in the title. Thanks.  
  > **username_0**: I added specific questions  such as 'same variables' and 'two distributions'. Thanks.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-07-20. Score: 3
  
  The KL divergence is just *not symmetric*, and so changing $q$ for $p$, and vice-versa, gives you a different behavior because the expectation is computed on a different distribution.
  
  * In the first plot, you see the **forward KL**, $KL(q\ \|\ p)$, that encourages the approximating distribution $q(z)$ (the one you topically learn) to cover the mode of $p(z)$: this is called *mode-seeking* behavior, which tries to match the most-likely point of the distribution $p$.
  * The other plot shows, instead, the **reverse KL**, $KL(p\ \|\ q)$, that tries to match the support of $p(z)$, however loosing the mode in the approximating distribution.
  
  To recap, the forward KL tries to put probability mass in the area of the distribution where the probability of samples is higher (in this example, the area where the std is 1), while the reverse KL tries just to cover all the distribution's support placing some non-zero probability everywhere: that's why the second, bigger, red circle covers the most of the green contours.
  
  **Update:**
  
  > Which is the same variables?
  
  The (random) variables $z\_1$ and $z\_2$ define the "space" of values that you can observe, e.g. you're given a dataset of samples taken from $p(z)$, i.e. the green distribution, where $z \in \mathbb [0,1]^2$. Basically, the approximating distribution, $q(z)$, is defined on the *same* space where the variables $z\_1$ and $z\_2$ lie.
  
  > Which is the two distributions?
  
  I think Bishop assumes $q(z)$ to be defined as a *mean field* approximation, which should correspond to $q(z) = \prod\_{i\in\{1,2\}} q(z\_i; \mu\_i, \sigma)$ where $\mu\_i$ and $\sigma$ are the parameters you want to learn to approximate $p(z)$. Basically, you consider two tractable distributions, e.g. a Gaussian, and learn the parameters of each "simple" distribution. Then, to get the final, more complex, distribution you perform the mean field assumption which is about doing a product among each $q\_i(z\_i)$. Also I put $\sigma$ and not $\sigma\_i$ since from the figures the variance seems to be independent from $z\_i$. For example, the mean field assumption does not enable you to learn the correlation between each $z\_i$, so this may explain the same variance across dimensions.
  
  > What is the product? Does it mean the probability density function?
  
  I think a partial answer is above. Anyway, is a product over the pdf of each $p\_i$ and $q\_i$ to yield, respectively, $p(z)$ and $q(z)$. But in the case of $p(z)$ you also account for the
  correlations when doing the product, instead in $q(z)$ independence is assumed.
  
  In the end, you can define $p(z) = \mathcal N(\mu,\Sigma)$ where $\mu=[0.5,0.5]$ and $\Sigma=\begin{bmatrix} \sigma\_{11} & \sigma\_{12} \\ \sigma\_{21} & \sigma\_{22} \end{bmatrix}$, where each $\sigma\_{ij}$ (for $i=1,2$ and $j=1,2$) tells you the correlation between variable $z\_i$ and $z\_j$ that has the effect of shrinking or expanding and rotating (more importantly) the green Gaussian. Instead, $q(z)=\prod\_i \mathcal N(\hat\mu\_i,\hat\sigma\_i)$, since is an approximation, has no such capabilities (e.g., rotation) and so you get a perfect circle, smaller or larger according to the KL you minimize to learn the parameters: $\hat\mu\_1 = \hat\mu\_2 = 0.5$ and $\hat\sigma\_1 = \hat\sigma\_2 = \hat \sigma$ or $\hat\Sigma=\begin{bmatrix} \hat\sigma & 0 \\ 0 & \hat\sigma \end{bmatrix}$.
  
  > **Comments:**
  > **username_2**: @username_0 I'm glad the answer helped you. I've edited it trying to answer the other questions too. Have a look and let me know!  
  > **username_0**: Thank you. It's a bit complicated, so I'll try to read it slowly!  
  
  ---
  
  