Answers.Body: <p>No, the word vectors are not one-hot encodings. Yes, they are learned. </p>
  
  <p>The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are <span class="math-container">$d$</span>-dimensional vectors, then each word's position in this vector space <strong>says something about what that word means</strong>. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of <a href="https://en.wikipedia.org/wiki/Distributional_semantics" rel="nofollow noreferrer">distributional similarity</a>.</p>
  
  <p>The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.</p>
  
  <p>A naive objective function to maximize for word2vec is
  <span class="math-container">$$J = \prod_{t=1}^L \prod_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}_{t+j}|\textbf{v}_t)$$</span></p>
  
  <p>where <span class="math-container">$L$</span> is the length of your corpus, <span class="math-container">$m$</span> is the "radius" from each center word you want to consider, <span class="math-container">$\textbf{u}_{t+j}$</span> is an outside word, and <span class="math-container">$\textbf{v}_t$</span> is a center word. </p>
  
  <p>If we let <span class="math-container">$p(\textbf{u}_{t+j}|\textbf{v}_t)$</span> be the softmax distribution, then maximizing <span class="math-container">$J$</span> means maximizing the inner product <span class="math-container">$\textbf{u}_{t+j}^T\textbf{v}_t$</span> in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.</p>
  
  <p><a href="https://www.youtube.com/watch?v=ERibwqs9p38" rel="nofollow noreferrer">This lecture</a> from Stanford's CS224N goes into more detail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T19:51:48.480Z
Answers.Text: They're asking if training the model involves learning the "word vectors" u and v. These vectors are the parameters of the model. We don't do inference from word2vec, but the resulting word vectors are used to encode input text for some other NLP tasks. I'll edit the question to add more context.
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T20:55:41.437Z
Answers.Text: Word2vec is like an autoencoder in that it learns useful encodings of inputs in an unsupervised way. It is not, however, a neural network. I also meant that we are not interested, ultimately, in the final output distribution of word2vec. We do not infer outputs of the model (distributions over likely nearby words) on new data. As in autoencoders, we may infer from a model that makes use of the learned encodings, but we do not typically infer from the model itself. Maybe I'm thinking of inference the wrong way, though. [This](https://bit.ly/2G7WlOV) is roughly my idea of inference.
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-13T08:38:28.710Z
Answers.LastActivityDate: 2019-03-13T08:52:55.447Z
Answers.LastEditDate: 2019-03-13T08:52:55.447Z
Answers.UserDisplayName: Philip Raeisghasem
Body: <p>In <a href="https://en.wikipedia.org/wiki/Word2vec" rel="nofollow noreferrer">word2vec</a>, the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word <span class="math-container">$c$</span> in the corpus, the model outputs the probability distribution <span class="math-container">$P(O=o|C=c)$</span> of how likely each other word <span class="math-container">$o$</span> in the vocabulary is to be within a certain number of words away from <span class="math-container">$c$</span>. We call <span class="math-container">$c$</span> the "center word" and <span class="math-container">$o$</span> the "outside word".</p>
  
  <p>We choose the softmax distribution as the output of our model: <span class="math-container">$$P(O=o|C=c) = \frac{\exp(\textbf{u}_{0}^{T} \textbf{v}_{c})}{\sum_{w \in \text{Vocab}} \exp(\textbf{u}_{w}^{T} \textbf{v}_c)}$$</span></p>
  
  <p>where <span class="math-container">$\textbf{u}_0$</span> and <span class="math-container">$\textbf{v}_c$</span> are vectors that represent the outside and center words respectively. </p>
  
  <blockquote>
    <p><strong>Question.</strong> What do the vectors  <span class="math-container">$\textbf{u}_0$</span> and <span class="math-container">$\textbf{v}_c$</span> look like? Are they just one-hot-encodings? Do we need to learn them
    too? Why is this useful?</p>
  </blockquote>
  
Tags: <natural-language-processing><word2vec><word-embedding>
Title: What do the vectors of the center and outside word look like in word2vec?
UserDisplayName: aiguy123
ThreadText: # What do the vectors of the center and outside word look like in word2vec?
  
  (Asked by: username_0 on 2019-03-12)
  
  In [word2vec](https://en.wikipedia.org/wiki/Word2vec), the task is to learn to predict which words are most likely to be near each other in some long corpus of text. For each word $c$ in the corpus, the model outputs the probability distribution $P(O=o|C=c)$ of how likely each other word $o$ in the vocabulary is to be within a certain number of words away from $c$. We call $c$ the "center word" and $o$ the "outside word".
  
  We choose the softmax distribution as the output of our model: $$P(O=o|C=c) = \frac{\exp(\textbf{u}\_{0}^{T} \textbf{v}\_{c})}{\sum\_{w \in \text{Vocab}} \exp(\textbf{u}\_{w}^{T} \textbf{v}\_c)}$$
  
  where $\textbf{u}\_0$ and $\textbf{v}\_c$ are vectors that represent the outside and center words respectively.
  
  > **Question.** What do the vectors $\textbf{u}\_0$ and $\textbf{v}\_c$ look like? Are they just one-hot-encodings? Do we need to learn them
  > too? Why is this useful?
  
  **Tags:** natural-language-processing, word2vec, word-embedding
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-03-13. Score: 1
  
  No, the word vectors are not one-hot encodings. Yes, they are learned.
  
  The purpose of the word2vec model is actually to learn dense, semantically meaningful encodings for words. That is, if your words are $d$-dimensional vectors, then each word's position in this vector space **says something about what that word means**. This is because word2vec learns to represent words in similar ways if they are frequently close together in your corpus. It implements the idea of [distributional similarity](https://en.wikipedia.org/wiki/Distributional_semantics).
  
  The task of predicting an "outside word" given a "center word" accomplishes all of this in an indirect way.
  
  A naive objective function to maximize for word2vec is
  $$J = \prod\_{t=1}^L \prod\_{-m \leq j \leq m\\ \quad j\neq 0} p(\textbf{u}\_{t+j}|\textbf{v}\_t)$$
  
  where $L$ is the length of your corpus, $m$ is the "radius" from each center word you want to consider, $\textbf{u}\_{t+j}$ is an outside word, and $\textbf{v}\_t$ is a center word.
  
  If we let $p(\textbf{u}\_{t+j}|\textbf{v}\_t)$ be the softmax distribution, then maximizing $J$ means maximizing the inner product $\textbf{u}\_{t+j}^T\textbf{v}\_t$ in the softmax's numerator. Maximizing that inner product means making center words as close as possible to their neighboring words, giving you some semantically meaningful word vectors to use in your downstream NLP tasks.
  
  [This lecture](https://www.youtube.com/watch?v=ERibwqs9p38) from Stanford's CS224N goes into more detail.
  
  > **Comments:**
  > **username_1**: They're asking if training the model involves learning the "word vectors" u and v. These vectors are the parameters of the model. We don't do inference from word2vec, but the resulting word vectors are used to encode input text for some other NLP tasks. I'll edit the question to add more context.  
  > **username_1**: Word2vec is like an autoencoder in that it learns useful encodings of inputs in an unsupervised way. It is not, however, a neural network. I also meant that we are not interested, ultimately, in the final output distribution of word2vec. We do not infer outputs of the model (distributions over likely nearby words) on new data. As in autoencoders, we may infer from a model that makes use of the learned encodings, but we do not typically infer from the model itself. Maybe I'm thinking of inference the wrong way, though. [This](https://bit.ly/2G7WlOV) is roughly my idea of inference.  
  
  ---
  
  