Body: <p>I'm reading the notes <a href="https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes02-wordvecs2.pdf" rel="nofollow noreferrer">here</a> and have a doubt on page 2 ("Least squares objective" section). The probability of a word <span class="math-container">$j$</span> occurring in the context of word <span class="math-container">$i$</span> is <span class="math-container">$$Q_{ij}=\frac{\exp(u_j^Tv_i)}{\sum_{w=1}^W\exp(u_w^Tv_i)}$$</span></p>
  
  <p>The notes read:</p>
  
  <blockquote>
    <p>Training proceeds in an on-line, stochastic fashion, but the implied global cross-entropy loss can be calculated as <span class="math-container">$$J=-\sum_{i\in corpus}\sum_{j\in context(i)}\log Q_{ij}$$</span>
    As the same words <span class="math-container">$i$</span> and <span class="math-container">$j$</span> can occur multiple times in the corpus, it is more efficient to first group together the same values for <span class="math-container">$i$</span> and <span class="math-container">$j$</span>:
    <span class="math-container">$$J=-\sum_{i=1}^W\sum_{j=1}^WX_{ij}\log(Q_{ij})$$</span></p>
  </blockquote>
  
  <p>where <span class="math-container">$X_{ij}$</span> is the total number of times <span class="math-container">$j$</span> occurs in the context of <span class="math-container">$i$</span> and the value of co-occuring frequency is given by the co-occurence matrix <span class="math-container">$X$</span>. This much is clear. But then the author states that the denominator of <span class="math-container">$Q_{ij}$</span> is too expensive to compute, so the cross entropy loss won't work.</p>
  
  <blockquote>
    <p>Instead, we use a least square objective in which the normalization factors in <span class="math-container">$P$</span> and <span class="math-container">$Q$</span> are discarded:
    <span class="math-container">$$\hat J=\sum_{i=1}^W\sum_{j=1}^WX_i(\hat P_{ij}-\hat Q_{ij})^2$$</span>
    where <span class="math-container">$\hat P_{ij}=X_{ij}$</span> and <span class="math-container">$\hat Q_{ij}=\exp(u_j^Tv_i)$</span> are the unnormalized distributions.</p>
  </blockquote>
  
  <p><span class="math-container">$X_i=\sum_kX_{ik}$</span> is the number of times any word appears in the context of <span class="math-container">$i$</span>. I don't understand this part. <em>Why have we introduced <span class="math-container">$X_i$</span> out of nowhere? How is <span class="math-container">$\hat P_{ij}$</span> "unnormalized"? Is there a tradeoff in switching from softmax to MSE?</em> </p>
  
  <p>(As far as I know, softmax made total sense in skip gram because we were calculating scores corresponding to different words (discrete possibilities) and matching the predicted output to the actual word - similar to a classification problem, so softmax makes sense.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-17T17:20:28.320Z
Comments.Text: In the notes, the author specifically says " it requires the distribution Q to be properly normalized, which involves the expensive summation over the entire vocabulary", so I think you should also have said this, because it was not clear what the _denominator_ is in your context. So, the denominator is the normalization factor.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-17T17:25:40.920Z
Comments.Text: I think I have the answer to your question: _How is $\hat P_{ij}$ "unnormalized"?_. In section 1.2 of the notes, the author defines $P_{ij}$ as the fraction between $X_{ij}$ and $X_i$, so I suppose that $\hat P_{ij}$ is just $X_{ij}$ (only the numerator), which is _number of times word $j$ occur in
  the context of word $i$_.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-17T17:36:40.287Z
Comments.Text: @nbro: Ah you're absolutely right. Careless of me not to see that. But I'm still not sure why $X_i$ comes in
Comments.UserDisplayName: Shirish
Tags: <natural-language-processing><word-embedding><glove>
Title: Doubt on formulating cost function for GloVe
UserDisplayName: Shirish
ThreadText: # Doubt on formulating cost function for GloVe
  
  (Asked by: username_0 on 2019-09-17)
  
  I'm reading the notes [here](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes02-wordvecs2.pdf) and have a doubt on page 2 ("Least squares objective" section). The probability of a word $j$ occurring in the context of word $i$ is $$Q\_{ij}=\frac{\exp(u\_j^Tv\_i)}{\sum\_{w=1}^W\exp(u\_w^Tv\_i)}$$
  
  The notes read:
  
  > Training proceeds in an on-line, stochastic fashion, but the implied global cross-entropy loss can be calculated as $$J=-\sum\_{i\in corpus}\sum\_{j\in context(i)}\log Q\_{ij}$$
  > As the same words $i$ and $j$ can occur multiple times in the corpus, it is more efficient to first group together the same values for $i$ and $j$:
  > $$J=-\sum\_{i=1}^W\sum\_{j=1}^WX\_{ij}\log(Q\_{ij})$$
  
  where $X\_{ij}$ is the total number of times $j$ occurs in the context of $i$ and the value of co-occuring frequency is given by the co-occurence matrix $X$. This much is clear. But then the author states that the denominator of $Q\_{ij}$ is too expensive to compute, so the cross entropy loss won't work.
  
  > Instead, we use a least square objective in which the normalization factors in $P$ and $Q$ are discarded:
  > $$\hat J=\sum\_{i=1}^W\sum\_{j=1}^WX\_i(\hat P\_{ij}-\hat Q\_{ij})^2$$
  > where $\hat P\_{ij}=X\_{ij}$ and $\hat Q\_{ij}=\exp(u\_j^Tv\_i)$ are the unnormalized distributions.
  
  $X\_i=\sum\_kX\_{ik}$ is the number of times any word appears in the context of $i$. I don't understand this part. *Why have we introduced $X\_i$ out of nowhere? How is $\hat P\_{ij}$ "unnormalized"? Is there a tradeoff in switching from softmax to MSE?*
  
  (As far as I know, softmax made total sense in skip gram because we were calculating scores corresponding to different words (discrete possibilities) and matching the predicted output to the actual word - similar to a classification problem, so softmax makes sense.)
  
  **Tags:** natural-language-processing, word-embedding, glove
  
  > **Comments:**
  > **username_1**: In the notes, the author specifically says " it requires the distribution Q to be properly normalized, which involves the expensive summation over the entire vocabulary", so I think you should also have said this, because it was not clear what the _denominator_ is in your context. So, the denominator is the normalization factor.  
  > **username_1**: I think I have the answer to your question: _How is $\hat P_{ij}$ "unnormalized"?_. In section 1.2 of the notes, the author defines $P_{ij}$ as the fraction between $X_{ij}$ and $X_i$, so I suppose that $\hat P_{ij}$ is just $X_{ij}$ (only the numerator), which is _number of times word $j$ occur in the context of word $i$_.  
  > **username_0**: @username_1: Ah you're absolutely right. Careless of me not to see that. But I'm still not sure why $X_i$ comes in  
  
  ---
  
  ## 0 answers
  
  