Answers.Body: <p>Almost, but no. When you maximize that objective function, you do so by adjusting the parameters <span class="math-container">$\phi$</span> and <span class="math-container">$\theta$</span>. After you're done with training, you can use your word embeddings for other NLP tasks. You don't, however, do any prediction directly from the skip-gram model. </p>
  
  <p>To maximize the first term, co-occuring words must have large inner products. That is, they must be "similar". To maximize the second term**, the randomly sampled words must have a small inner product with <span class="math-container">$\phi_u$</span>. That is, they must be "dissimilar" to <span class="math-container">$\phi_u$</span>. Moving these word embeddings around in the vector space to make some words similar and others not is the only thing that happens during skip-gram training. 
  <span class="math-container">$$$$</span></p>
  
  <p>** <span class="math-container">$\sigma(-x)=1-\sigma(x)$</span>, so maximizing <span class="math-container">$\sigma(-x)$</span> is minimizing <span class="math-container">$\sigma(x)$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T15:09:11.480Z
Answers.Text: But isn't the idea of using Skip-Gram to bypass the computational complexity of the log-likelihood maximization? So instead of predicting a word given another word, we are predicting whether certain words appear together or not (implicitly)? See 9:43 of the video.
Answers.UserDisplayName: naturalguy_12
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T09:00:57.173Z
Answers.Text: No. But that idea is similar to one behind the [GloVe method](https://nlp.stanford.edu/pubs/glove.pdf) of creating word embeddings. Also, I'm not really a fan of the video you keep linking. Higher quality videos from Stanford's 224d class are [here](https://www.youtube.com/watch?v=ERibwqs9p38&list=PL3FW7Lu3i5Jsnh1rnUwq_TcylNr7EkRe6&index=2).
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-25T16:28:02.303Z
Answers.Text: Why aren't you a fan of the video?
Answers.UserDisplayName: naturalguy_12
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-20T10:55:55.553Z
Answers.LastActivityDate: 2019-03-20T10:55:55.553Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>Suppose we want to predict context words <span class="math-container">$w_{i-h}, \dots, w_{i+h}$</span> given a target word <span class="math-container">$w_i$</span> for a window size <span class="math-container">$h$</span> around the target word <span class="math-container">$w_i$</span>. We can represent this as: <span class="math-container">$$p(w_{i-h}, \dots, w_{i+h}|w_i) = \prod_{-h \leq k \leq h, \ k \neq 0} p(w_{i+k}|w_i)$$</span> where we model the probabilities of a word <span class="math-container">$u$</span> given another word <span class="math-container">$v$</span> as <span class="math-container">$$p(u|v) = \frac{\exp(\left&lt;\phi_u, \theta_v \right&gt;)}{\sum_{u' \in W} \exp(\left&lt;\phi_{u'}, \theta_v \right&gt;)}$$</span> where <span class="math-container">$\phi_u, \theta_v$</span> are some vector representations for words <span class="math-container">$u$</span> and <span class="math-container">$v$</span> respectively and <span class="math-container">$\left&lt;\phi_u, \theta_v \right&gt;$</span> is the dot product between these vector representations (which represents some sort of similarity between the words) and <span class="math-container">$W$</span> is a matrix of all the words.</p>
  
  <p>In Skip-Gram Negative Sampling, we want to learn the embeddings <span class="math-container">$\phi_u, \theta_v$</span> that maximize the following: <span class="math-container">$$\sum_{u \in W} \sum_{v \in W} n_{uv} \log \sigma(\left&lt;\phi_u, \theta_v \right&gt;) +k \mathbb{E}_{\bar{v}} \log \sigma(-\left&lt;\phi_u, \theta_{\bar{v}} \right&gt;)$$</span></p>
  
  <blockquote>
    <p><strong>Question.</strong> How exactly does this work? For example, suppose <span class="math-container">$k=5$</span>, the target word <span class="math-container">$w_i$</span> is <span class="math-container">$\text{apple}$</span> and we want to find
    <span class="math-container">$p(\text{pie}| \text{apple})$</span>. Let <span class="math-container">$n_{uv} = 10$</span> (number of times pie
    co-occurs with apple). Then we sample <span class="math-container">$5$</span> random words <span class="math-container">$\bar{v}$</span> that
    did not occur with <span class="math-container">$\text{apple}$</span> and whichever term in the sum is
    bigger is the one we predict? For example, if the first term in the
    sum is larger than the second term then we would predict that
    <span class="math-container">$p(\text{pie}| \text{apple}) \approx 1$</span>? Otherwise we predict that
    <span class="math-container">$p(\text{pie}| \text{apple}) \approx 0$</span>? Is this the correct
    intuition?</p>
  </blockquote>
  
  <p><strong>Source.</strong> <a href="https://www.coursera.org/learn/language-processing/lecture/A4eQD/explicit-and-implicit-matrix-factorization" rel="nofollow noreferrer">Here</a> at around the 10:05 mark.</p>
  
Tags: <natural-language-processing><word2vec><word-embedding>
Title: Skip-Gram Model Training
UserDisplayName: naturalguy_12
ThreadText: # Skip-Gram Model Training
  
  (Asked by: username_0 on 2019-03-19)
  
  Suppose we want to predict context words $w\_{i-h}, \dots, w\_{i+h}$ given a target word $w\_i$ for a window size $h$ around the target word $w\_i$. We can represent this as: $$p(w\_{i-h}, \dots, w\_{i+h}|w\_i) = \prod\_{-h \leq k \leq h, \ k \neq 0} p(w\_{i+k}|w\_i)$$ where we model the probabilities of a word $u$ given another word $v$ as $$p(u|v) = \frac{\exp(\left<\phi\_u, \theta\_v \right>)}{\sum\_{u' \in W} \exp(\left<\phi\_{u'}, \theta\_v \right>)}$$ where $\phi\_u, \theta\_v$ are some vector representations for words $u$ and $v$ respectively and $\left<\phi\_u, \theta\_v \right>$ is the dot product between these vector representations (which represents some sort of similarity between the words) and $W$ is a matrix of all the words.
  
  In Skip-Gram Negative Sampling, we want to learn the embeddings $\phi\_u, \theta\_v$ that maximize the following: $$\sum\_{u \in W} \sum\_{v \in W} n\_{uv} \log \sigma(\left<\phi\_u, \theta\_v \right>) +k \mathbb{E}\_{\bar{v}} \log \sigma(-\left<\phi\_u, \theta\_{\bar{v}} \right>)$$
  
  > **Question.** How exactly does this work? For example, suppose $k=5$, the target word $w\_i$ is $\text{apple}$ and we want to find
  > $p(\text{pie}| \text{apple})$. Let $n\_{uv} = 10$ (number of times pie
  > co-occurs with apple). Then we sample $5$ random words $\bar{v}$ that
  > did not occur with $\text{apple}$ and whichever term in the sum is
  > bigger is the one we predict? For example, if the first term in the
  > sum is larger than the second term then we would predict that
  > $p(\text{pie}| \text{apple}) \approx 1$? Otherwise we predict that
  > $p(\text{pie}| \text{apple}) \approx 0$? Is this the correct
  > intuition?
  
  **Source.** [Here](https://www.coursera.org/learn/language-processing/lecture/A4eQD/explicit-and-implicit-matrix-factorization) at around the 10:05 mark.
  
  **Tags:** natural-language-processing, word2vec, word-embedding
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-20. Score: 1
  
  Almost, but no. When you maximize that objective function, you do so by adjusting the parameters $\phi$ and $\theta$. After you're done with training, you can use your word embeddings for other NLP tasks. You don't, however, do any prediction directly from the skip-gram model.
  
  To maximize the first term, co-occuring words must have large inner products. That is, they must be "similar". To maximize the second term\*\*, the randomly sampled words must have a small inner product with $\phi\_u$. That is, they must be "dissimilar" to $\phi\_u$. Moving these word embeddings around in the vector space to make some words similar and others not is the only thing that happens during skip-gram training.
  $$$$
  
  \*\* $\sigma(-x)=1-\sigma(x)$, so maximizing $\sigma(-x)$ is minimizing $\sigma(x)$
  
  > **Comments:**
  > **username_0**: But isn't the idea of using Skip-Gram to bypass the computational complexity of the log-likelihood maximization? So instead of predicting a word given another word, we are predicting whether certain words appear together or not (implicitly)? See 9:43 of the video.  
  > **username_1**: No. But that idea is similar to one behind the [GloVe method](https://nlp.stanford.edu/pubs/glove.pdf) of creating word embeddings. Also, I'm not really a fan of the video you keep linking. Higher quality videos from Stanford's 224d class are [here](https://www.youtube.com/watch?v=ERibwqs9p38&list=PL3FW7Lu3i5Jsnh1rnUwq_TcylNr7EkRe6&index=2).  
  > **username_0**: Why aren't you a fan of the video?  
  
  ---
  
  