Answers.Body: <p>Note: mutual information is typically expressed as a log value (usually <span class="math-container">$log_2$</span>, as it's related to information), which makes them easier to compare -- you then don't have to worry about large exponential expressions with negative exponents.</p>
  <p>The reason for the bias is in the distributional properties of words. A rare word will have a small frequency of occurrence (by definition), so multiplied with a more common word, the denominator will be fairly small. But the numerator will also be very small, as there aren't that many opportunities for the rare word to occur near the more common word.</p>
  <p>However, while with two common words the nominator will be much higher (more co-occurrences), the denominator will now — compared to the rare/common instance — be several orders of magnitude larger. Thus the overall value will be smaller.</p>
  <p>Rare co-occurrences are overly dependent on chance, as the nominator can easily fluctuate randomly (say, 1 or 2 co-occurences), and with a comparatively smaller denominator that can make a big difference.</p>
  <p>In linguistics, you would generally ignore mutual information values above a certain threshold, as the values are just too unreliable to be meaningful. In fact, when I was still working in academia, mutual information was increasingly replaced by other metrics, such as log-likelihood, which were more robust.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T23:32:26.780Z
Answers.Text: Although it is okay to understand. If possible, please keep in brackets which one you are referring as word and which one as context.
Answers.UserDisplayName: hanugm
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-14T07:16:30.387Z
Answers.LastActivityDate: 2021-06-14T07:16:30.387Z
Answers.UserDisplayName: Oliver Mason
Body: <p>Consider the following formulation for pointwise mutual information (PMI):</p>
  <p><span class="math-container">$$\text{PMI}(w, c) = \dfrac{p(w, c)}{p(w)p(c)}$$</span></p>
  <p>Suppose there are <span class="math-container">$W$</span> words with <span class="math-container">$C$</span> context words. Then one can write in terms of frequency that</p>
  <p><span class="math-container">$$\text{PMI}(w, c) = \dfrac{\sum\limits_{i = 1}^{W} \sum\limits_{j = 1}^{C} f_{ij} }{\sum\limits_{i = 1}^{W}f_i \sum\limits_{j = 1}^{C} f_j} $$</span></p>
  <p>I am going to calculate <span class="math-container">$\text{PMI}(w, c)$</span> for two different words and contexts based on the following table. The table is taken from fig 6.10 of <a href="https://web.stanford.edu/%7Ejurafsky/slp3/6.pdf" rel="nofollow noreferrer">this</a> book.</p>
  <p><a href="https://i.sstatic.net/XNoXK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/XNoXK.png" alt="enter image description here" /></a></p>
  <p>I calculated PMI for all pairs and tabulated below.</p>
  <p><span class="math-container">$$\begin{array}{|c|c|c|} \hline
      &amp; \text{computer} &amp; \text{data} &amp; \text{result} &amp; \text{pie} &amp;  \text{sugar} \\  \hline
    \text{cherry} &amp; 8.2 \times 10^{-7} &amp; 2.9 \times 10^{-6} &amp; 3.9 \times 10^{-5} &amp; 1.7 \times 10^{-3} &amp; 8.4 \times 10^{-4} \\ \hline
  \text{strawberry} &amp; 0 &amp; 0 &amp; 2.6 \times 10^{-5} &amp; 1.4 \times 10^{-3} &amp; 3.8 \times 10^{-3} \\ \hline
  \text{digital} &amp; 9.6 \times 10^{-5} &amp; 8.6 \times 10^{-5} &amp; 5.2  \times 10^{-5} &amp; 2.8 \times 10^{-6} &amp; 1.9 \times 10^{-5} \\ \hline
  \text{information} &amp; 8.6 \times 10^{-5} &amp; 9.1 \times 10^{-5} &amp; 1.03  \times 10^{-4} &amp; 1.2 \times 10^{-6}&amp; 2.7 \times 10^{-5}\\ \hline
  \end{array}$$</span></p>
  <p>Based on the above values, we can also notice the following fact:</p>
  <blockquote>
  <p>PMI has the problem of being biased toward infrequent events; very rare words tend to have very high PMI values.</p>
  </blockquote>
  <p>However, it's unclear to me how this apparent behaviour is related to the mathematical formulation of the PMI above.</p>
  <p>How do we understand the fact quoted above from the fractional form of PMI given by the equations above?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-14T00:33:23.463Z
Comments.Text: "But, it is not intuitive for me to analyze the fact using the analytical form of PMI given on the top of this question." this seems to be worded quite confusingly.
Comments.UserDisplayName: The Pointer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-14T01:00:26.293Z
Comments.Text: @ThePointer I forget how to write equation numbers, so I told like that.
Comments.UserDisplayName: hanugm
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-14T01:34:54.467Z
Comments.Text: @hanugm I think that the confusing part in that sentence is that " analyze the fact using", because it's not super clear what "fact" you're referring to. Maybe you could rewrite that sentence as follows "However, it's unclear to me  how this apparent behaviour is related to the mathematical formulation of the PMI above".
Comments.UserDisplayName: nbro
Tags: <natural-language-processing><intuition>
Title: How do very rare words tend to have very high PMI values?
UserDisplayName: hanugm
ThreadText: # How do very rare words tend to have very high PMI values?
  
  (Asked by: username_0 on 2021-06-14)
  
  Consider the following formulation for pointwise mutual information (PMI):
  
  $$\text{PMI}(w, c) = \dfrac{p(w, c)}{p(w)p(c)}$$
  
  Suppose there are $W$ words with $C$ context words. Then one can write in terms of frequency that
  
  $$\text{PMI}(w, c) = \dfrac{\sum\limits\_{i = 1}^{W} \sum\limits\_{j = 1}^{C} f\_{ij} }{\sum\limits\_{i = 1}^{W}f\_i \sum\limits\_{j = 1}^{C} f\_j} $$
  
  I am going to calculate $\text{PMI}(w, c)$ for two different words and contexts based on the following table. The table is taken from fig 6.10 of [this](https://web.stanford.edu/%7Ejurafsky/slp3/6.pdf) book.
  
  [![enter image description here](https://i.sstatic.net/XNoXK.png)](https://i.sstatic.net/XNoXK.png)
  
  I calculated PMI for all pairs and tabulated below.
  
  $$\begin{array}{|c|c|c|} \hline
  & \text{computer} & \text{data} & \text{result} & \text{pie} & \text{sugar} \\ \hline
  \text{cherry} & 8.2 \times 10^{-7} & 2.9 \times 10^{-6} & 3.9 \times 10^{-5} & 1.7 \times 10^{-3} & 8.4 \times 10^{-4} \\ \hline
  \text{strawberry} & 0 & 0 & 2.6 \times 10^{-5} & 1.4 \times 10^{-3} & 3.8 \times 10^{-3} \\ \hline
  \text{digital} & 9.6 \times 10^{-5} & 8.6 \times 10^{-5} & 5.2 \times 10^{-5} & 2.8 \times 10^{-6} & 1.9 \times 10^{-5} \\ \hline
  \text{information} & 8.6 \times 10^{-5} & 9.1 \times 10^{-5} & 1.03 \times 10^{-4} & 1.2 \times 10^{-6}& 2.7 \times 10^{-5}\\ \hline
  \end{array}$$
  
  Based on the above values, we can also notice the following fact:
  
  > PMI has the problem of being biased toward infrequent events; very rare words tend to have very high PMI values.
  
  However, it's unclear to me how this apparent behaviour is related to the mathematical formulation of the PMI above.
  
  How do we understand the fact quoted above from the fractional form of PMI given by the equations above?
  
  **Tags:** natural-language-processing, intuition
  
  > **Comments:**
  > **username_1**: "But, it is not intuitive for me to analyze the fact using the analytical form of PMI given on the top of this question." this seems to be worded quite confusingly.  
  > **username_0**: @username_1 I forget how to write equation numbers, so I told like that.  
  > **username_2**: @username_0 I think that the confusing part in that sentence is that " analyze the fact using", because it's not super clear what "fact" you're referring to. Maybe you could rewrite that sentence as follows "However, it's unclear to me  how this apparent behaviour is related to the mathematical formulation of the PMI above".  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-06-14. Score: 1
  
  Note: mutual information is typically expressed as a log value (usually $log\_2$, as it's related to information), which makes them easier to compare -- you then don't have to worry about large exponential expressions with negative exponents.
  
  The reason for the bias is in the distributional properties of words. A rare word will have a small frequency of occurrence (by definition), so multiplied with a more common word, the denominator will be fairly small. But the numerator will also be very small, as there aren't that many opportunities for the rare word to occur near the more common word.
  
  However, while with two common words the nominator will be much higher (more co-occurrences), the denominator will now — compared to the rare/common instance — be several orders of magnitude larger. Thus the overall value will be smaller.
  
  Rare co-occurrences are overly dependent on chance, as the nominator can easily fluctuate randomly (say, 1 or 2 co-occurences), and with a comparatively smaller denominator that can make a big difference.
  
  In linguistics, you would generally ignore mutual information values above a certain threshold, as the values are just too unreliable to be meaningful. In fact, when I was still working in academia, mutual information was increasingly replaced by other metrics, such as log-likelihood, which were more robust.
  
  > **Comments:**
  > **username_0**: Although it is okay to understand. If possible, please keep in brackets which one you are referring as word and which one as context.  
  
  ---
  
  