Body: <p>In his article <a href="https://arxiv.org/abs/1411.2738" rel="nofollow noreferrer">word2vec Parameter Learning Explained</a> Xin Rong says (page 7):</p>
  <blockquote>
  <p>Each output is computed using the same hidden-&gt;output matrix:
  <span class="math-container">$$ p(w_{c,j} = w_{O.c}|w_I)=y_{c,j}=\frac{exp(u_{c,j})}{\sum_{j'=1}^{V}exp(u_{j'})} \ \ \ \  (25) $$</span></p>
  </blockquote>
  <p>Looking into the <a href="https://github.com/dav/word2vec" rel="nofollow noreferrer">word2vec source code</a> I don’t see any “panels” or “output layers”. With this most of the terms in the equation above in totally unclear for me. Could you please help me realizing how this mathematical description intended to work (I understand the source code, but mathematical description is another question here).</p>
  <p>Am I missing something here? With the description above from the article, since “the output layer panels share the same weights”, how the results on output panels (even if they existed could be different)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-11T02:16:56.310Z
Comments.Text: The right hand side is called a softmax function, it turns a set of numbers, eg. [10, 3, 1, 0] into a probability distribution [0.99, 001, 0, 0] which always totals 1. It means the probability of the 1st token/word appearing next is 0.99, the probability of the 2nd token appearing next is 0.01, and there is no chance of the 3rd of 4th token appearing next.
Comments.UserDisplayName: James
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-11T12:52:22.913Z
Comments.Text: @James, thank you for the input. I know what Softmax is and more than that I know how hierarchical softmax works here. May be my wording of the question is vague, but I am concerned move with C-indexes here, since in the code which is described by this formula there are no panels at all. Meanwhile I noticed that the actual code uses "window" of words in the text so maybe the panels are "virtual" in the implementation. That is my question here. And if we have the same input word and same weights on the panels, how the outcome could be different. I know how it works in code, but the formula...
Comments.UserDisplayName: Damir Tenishev
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-11T17:37:01.077Z
Comments.Text: I skimmed through the pdf but the index notation seems a lot of work to wade into. Conceptually though as I understand it the skipgram is a simple tallying of surrounding words relative to the current word, eg. for sentence "am i missing something here", the skipgram for the word "missing" should read [0, 0.5, 0, 0.5, 0] for just this one sentence. The probabilities gets muddied where there are multiple occurences of the word "missing" throughout the learning text.
Comments.UserDisplayName: James
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-06-11T21:26:20.543Z
Comments.Text: @James, thanks again. And again, I know what is the skip-gram and how exactly it works. I was interested in one thing, how math describes the existing code to learn they way to cover things. The author of the paper here uses "panels" which are not represented explicitly in code which was my first concern. Then he says "the output layer panels share the same weights", if this is true than for the same input word they must produce the same output. Again, I know how this works (correctly in code), I am struggling with the "math" description of this.
Comments.UserDisplayName: Damir Tenishev
Tags: <word-embedding><embeddings><word2vec><skip-gram>
Title: Skip-Gram Model description in word2vec explanation article
UserDisplayName: Damir Tenishev
ThreadText: # Skip-Gram Model description in word2vec explanation article
  
  (Asked by: username_0 on 2024-06-08)
  
  In his article [word2vec Parameter Learning Explained](https://arxiv.org/abs/1411.2738) Xin Rong says (page 7):
  
  > Each output is computed using the same hidden->output matrix:
  > $$ p(w\_{c,j} = w\_{O.c}|w\_I)=y\_{c,j}=\frac{exp(u\_{c,j})}{\sum\_{j'=1}^{V}exp(u\_{j'})} \ \ \ \ (25) $$
  
  Looking into the [word2vec source code](https://github.com/dav/word2vec) I don’t see any “panels” or “output layers”. With this most of the terms in the equation above in totally unclear for me. Could you please help me realizing how this mathematical description intended to work (I understand the source code, but mathematical description is another question here).
  
  Am I missing something here? With the description above from the article, since “the output layer panels share the same weights”, how the results on output panels (even if they existed could be different)?
  
  **Tags:** word-embedding, embeddings, word2vec, skip-gram
  
  > **Comments:**
  > **username_1**: The right hand side is called a softmax function, it turns a set of numbers, eg. [10, 3, 1, 0] into a probability distribution [0.99, 001, 0, 0] which always totals 1. It means the probability of the 1st token/word appearing next is 0.99, the probability of the 2nd token appearing next is 0.01, and there is no chance of the 3rd of 4th token appearing next.  
  > **username_0**: @username_1, thank you for the input. I know what Softmax is and more than that I know how hierarchical softmax works here. May be my wording of the question is vague, but I am concerned move with C-indexes here, since in the code which is described by this formula there are no panels at all. Meanwhile I noticed that the actual code uses "window" of words in the text so maybe the panels are "virtual" in the implementation. That is my question here. And if we have the same input word and same weights on the panels, how the outcome could be different. I know how it works in code, but the formula...  
  > **username_1**: I skimmed through the pdf but the index notation seems a lot of work to wade into. Conceptually though as I understand it the skipgram is a simple tallying of surrounding words relative to the current word, eg. for sentence "am i missing something here", the skipgram for the word "missing" should read [0, 0.5, 0, 0.5, 0] for just this one sentence. The probabilities gets muddied where there are multiple occurences of the word "missing" throughout the learning text.  
  > **username_0**: @username_1, thanks again. And again, I know what is the skip-gram and how exactly it works. I was interested in one thing, how math describes the existing code to learn they way to cover things. The author of the paper here uses "panels" which are not represented explicitly in code which was my first concern. Then he says "the output layer panels share the same weights", if this is true than for the same input word they must produce the same output. Again, I know how this works (correctly in code), I am struggling with the "math" description of this.  
  
  ---
  
  ## 0 answers
  
  