Answers.Body: <p>The following figure from <a href="https://aegis4048.github.io/demystifying_neural_network_in_skip_gram_language_modeling" rel="nofollow noreferrer">this article</a> can be helpful:</p>
  <p><a href="https://i.sstatic.net/AsvbM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AsvbM.png" alt="enter image description here" /></a></p>
  <p>This figure represents &quot;Skip-Gram model structure. Current center word is 'passes'&quot;.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-18T10:51:41.473Z
Answers.LastActivityDate: 2020-10-18T10:51:41.473Z
Answers.UserDisplayName: OmG
Body: <p>I am a mathematics student who is learning NLP, so I have paid a high amount of attention on the mathematics used in the subject, but my interpretations may or may not be right sometimes. Please correct me if any of them are incorrect or do not make sense.</p>
  <p>I have learned CBOW and Skip-Gram models.</p>
  <p>I think I have understood the CBOW model, and here is my interpretation: First, we fix a number of neighbors of the unknown center word which we would like to predict; let the number be <span class="math-container">$m$</span>. We then input the original characteristic vectors (vectors of zeros and ones only) of those <span class="math-container">$2m$</span> context words. By multiplying those vectors by a matrix, we obtain <span class="math-container">$2m$</span> new vectors. Next, we take the average of those <span class="math-container">$2m$</span> vectors and this is our hidden layer, namely <span class="math-container">$v$</span>. We finally multiply <span class="math-container">$v$</span> with another matrix, and that is the &quot;empirical&quot; result.</p>
  <p>I tried to follow the logic to Skip-Gram similarly, but I have been stuck. I understand that Skip-Gram is kind of a &quot;reversal&quot; of CBOW, but the specific steps have given me a hard time. So, in Skip-Gram, we only have a center word, and based upon that we are trying to predict <span class="math-container">$2m$</span> context words. By similar steps, we obtain a hidden layer, which is again a vector. The final process also involves multiplication with a matrix, but I don't know how we can get <span class="math-container">$2m$</span> new vectors based upon one, unless we have <span class="math-container">$2m$</span> different matrices?</p>
  
Tags: <natural-language-processing><math><word2vec><cbow><skip-gram>
Title: Is my interpretation of the mathematics of the CBOW and Skip-Gram models correct?
UserDisplayName: Robert
ThreadText: # Is my interpretation of the mathematics of the CBOW and Skip-Gram models correct?
  
  (Asked by: username_0 on 2020-10-18)
  
  I am a mathematics student who is learning NLP, so I have paid a high amount of attention on the mathematics used in the subject, but my interpretations may or may not be right sometimes. Please correct me if any of them are incorrect or do not make sense.
  
  I have learned CBOW and Skip-Gram models.
  
  I think I have understood the CBOW model, and here is my interpretation: First, we fix a number of neighbors of the unknown center word which we would like to predict; let the number be $m$. We then input the original characteristic vectors (vectors of zeros and ones only) of those $2m$ context words. By multiplying those vectors by a matrix, we obtain $2m$ new vectors. Next, we take the average of those $2m$ vectors and this is our hidden layer, namely $v$. We finally multiply $v$ with another matrix, and that is the "empirical" result.
  
  I tried to follow the logic to Skip-Gram similarly, but I have been stuck. I understand that Skip-Gram is kind of a "reversal" of CBOW, but the specific steps have given me a hard time. So, in Skip-Gram, we only have a center word, and based upon that we are trying to predict $2m$ context words. By similar steps, we obtain a hidden layer, which is again a vector. The final process also involves multiplication with a matrix, but I don't know how we can get $2m$ new vectors based upon one, unless we have $2m$ different matrices?
  
  **Tags:** natural-language-processing, math, word2vec, cbow, skip-gram
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-18. Score: 1
  
  The following figure from [this article](https://aegis4048.github.io/demystifying_neural_network_in_skip_gram_language_modeling) can be helpful:
  
  [![enter image description here](https://i.sstatic.net/AsvbM.png)](https://i.sstatic.net/AsvbM.png)
  
  This figure represents "Skip-Gram model structure. Current center word is 'passes'".
  
  
  ---
  
  