Answers.Body: <p>Yes, you can think of mutual information as entropy carried by that information <span class="math-container">$Y$</span></p>
  <p>Say you have a dice <span class="math-container">$X$</span> and you want to predict the outcome, then your belief is a discrete uniform one with <span class="math-container">$p=1/6$</span> so the entropy is around 1.79</p>
  <p>Now, suppose I tell you that the outcome is <span class="math-container">$&lt;3$</span> (would be your <span class="math-container">$Y$</span>), now your belief is 50% on 1&amp;2 and 0 on the other outcomes, and your entropy is now 0.7</p>
  <p>As you can see, <span class="math-container">$Y$</span> carries a lot of entropy, because knowing it reduces a lot the uncertainty, thus having a big mutual information with <span class="math-container">$X$</span></p>
  <p>Obviously, you cannot have an information that increases entropy, as at most <span class="math-container">$X$</span> is conditional independent to <span class="math-container">$Y$</span>, and the two terms of the MI are equal, so the difference is 0</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-01T10:37:45.737Z
Answers.LastActivityDate: 2023-10-01T10:37:45.737Z
Answers.UserDisplayName: Alberto
Body: <p>I'm studying the chapter of Information theory from Haykin's deep learning book.</p>
  <p>It says Mutual Information between two continuous random variables <span class="math-container">$X,Y$</span> is defined in terms of the differential entropies <span class="math-container">$h(\cdot)$</span> as <span class="math-container">$I(X;Y)=h(X)-h(X|Y) = -\int_{-\infty}^{\infty}p_X(x)\log p_X(x) + \int _{-\infty}^{\infty} p_{X|Y}\log p_{X|Y}p(x|y)$</span>.</p>
  <p>Also, is it correct to interpret mutual information as a measure of &quot;reduction in uncertainty&quot;? I mean, if the mutual information <span class="math-container">$I(X;Y)$</span> is high, then there's gain in information of the random variable <span class="math-container">$X$</span> by observing <span class="math-container">$Y$</span>.</p>
  
Tags: <information-theory><uncertainty-quantification>
Title: How Mutual Information is related to uncertainty
UserDisplayName: piero
ThreadText: # How Mutual Information is related to uncertainty
  
  (Asked by: username_0 on 2023-10-01)
  
  I'm studying the chapter of Information theory from Haykin's deep learning book.
  
  It says Mutual Information between two continuous random variables $X,Y$ is defined in terms of the differential entropies $h(\cdot)$ as $I(X;Y)=h(X)-h(X|Y) = -\int\_{-\infty}^{\infty}p\_X(x)\log p\_X(x) + \int \_{-\infty}^{\infty} p\_{X|Y}\log p\_{X|Y}p(x|y)$.
  
  Also, is it correct to interpret mutual information as a measure of "reduction in uncertainty"? I mean, if the mutual information $I(X;Y)$ is high, then there's gain in information of the random variable $X$ by observing $Y$.
  
  **Tags:** information-theory, uncertainty-quantification
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-10-01. Score: 1 | Accepted answer
  
  Yes, you can think of mutual information as entropy carried by that information $Y$
  
  Say you have a dice $X$ and you want to predict the outcome, then your belief is a discrete uniform one with $p=1/6$ so the entropy is around 1.79
  
  Now, suppose I tell you that the outcome is $<3$ (would be your $Y$), now your belief is 50% on 1&2 and 0 on the other outcomes, and your entropy is now 0.7
  
  As you can see, $Y$ carries a lot of entropy, because knowing it reduces a lot the uncertainty, thus having a big mutual information with $X$
  
  Obviously, you cannot have an information that increases entropy, as at most $X$ is conditional independent to $Y$, and the two terms of the MI are equal, so the difference is 0
  
  
  ---
  
  