Answers.Body: <p>Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words.</p>
  
  <p>lets assume two words <span class="math-container">$e_i$</span> and <span class="math-container">$e_j$</span> pop up in the exact same context of some word <span class="math-container">$e_k$</span> with 3 other context words as well. An example would be:</p>
  
  <ol>
  <li>He <em>leaped</em> <strong>over</strong> the truck</li>
  <li>He <em>jumped</em> <strong>over</strong> the truck</li>
  </ol>
  
  <p>Where the italics represent the words with similar meanings but the bolded words above are the ones being predicted/skipped in CBOW. Lets now show this.</p>
  
  <p>Let the rest of the context be denoted <span class="math-container">$\{e_r\}_r$</span> and the skipped word as <span class="math-container">$e_s$</span> and so the loss will try to minimize both <span class="math-container">$-e_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_i + \sum_re_r)]))$</span> and <span class="math-container">$-e_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_j + \sum_re_r)]))$</span>.   </p>
  
  <p>Assuming a long run withlarge enough batch-size (so ignoring catastrophic forgetting) we can essentially say it will be minimizing<br>
  <span class="math-container">$$-e_s^T [log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_i + \sum_re_r)])) +\\ log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e_j + \sum_re_r)]))]$$</span><br>
  Now let <span class="math-container">$\mathcal{U}\mathcal{V}\frac{1}{R+1}\sum_re_r$</span> be denoted as <span class="math-container">$c$</span>, <span class="math-container">$\mathcal{U}\mathcal{V}\frac{1}{R+1}e_i$</span> as <span class="math-container">$\hat u_i$</span>, and <span class="math-container">$\mathcal{U}\mathcal{V}\frac{1}{R+1}e_j$</span> as <span class="math-container">$\hat u_j$</span>. So we have to minimize<br>
  <span class="math-container">$$-e_s^T[log(softmax(\hat u_i+c)) + log(softmax(\hat u_j+c))]$$</span><br>
  or equivalently<br>
  <span class="math-container">$$-e_s^T\ log(softmax(\hat u_i+c) * softmax(\hat u_j+c))$$</span><br>
  As we know in crossentropy, we get our critical point when the inside of the log here would equal <span class="math-container">$e_s$</span>, which is only possible if both <span class="math-container">$softmax(\hat u_i+c)$</span> and <span class="math-container">$softmax(\hat u_j+c)$</span> were to be equal to <span class="math-container">$e_s$</span>. softmax is non-invertible and is invariant under addition only, therefore to achieve this we would need <span class="math-container">$\hat u_i + c$</span> to equal <span class="math-container">$\hat u_j + c + K$</span> where <span class="math-container">$K$</span> is just some arbitrary constant vector (<span class="math-container">$K=[k,k,k,...]$</span>. Subtracting <span class="math-container">$c$</span> and multiplying by <span class="math-container">$R+1$</span> we make our model want  </p>
  
  <p><span class="math-container">$$\begin{align}
  u_i &amp;= u_j + K \\
  \implies K &amp;= \mathcal{U}\mathcal{V}(e_i -e_j) \\
  \end{align}$$</span></p>
  
  <p>This would mean for all common words, we see this activity that <span class="math-container">$u_{i-j}$</span> would be a constant, but this is difficult because that means a <span class="math-container">$\delta$</span> must exist such that <span class="math-container">$\mathcal{U}\delta = k*\vec 1$</span> and that all similar word pairs indexed by <span class="math-container">$(a,b)$</span> would have <span class="math-container">$v_a \approx v_b + const*\delta$</span> (because <span class="math-container">$u_a = u_b + K \implies v_b = v_a + \mathcal{U}^{-1}K$</span> and we denote <span class="math-container">$\delta = \mathcal{U}^{-1}K$</span>). This constraint is heavy, tough to learn, and would also be difficult to process on multiple word senses that would appear in the vocabulary. This is because if it were to learn this, it enforces constraints on the images of <span class="math-container">$\mathcal{U}$</span> and <span class="math-container">$\mathcal{V}$</span> along with creating word vectors with highly different magnitudes making the learning process difficult. This would indicate that this constant would be low in practice. Therefore we would get <span class="math-container">$v_i \approx v_j$</span>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T11:57:51.840Z
Answers.Text: Thanks for the nice answer! I have a few doubts: there's a couple of assumptions underlying the whole treatment - 1. The central (skipped) word as well as all **other** words in the context are **exactly** the same. 2. The definition of "similar" words assumed by your answer is ones that occur in the context of the **same** word (in this case, "over"). The definition is _not_: similar words are those that occur in same/similar contexts. Am I right that these are the assumptions? In their current form, aren't they quite restrictive/unlikely?
Answers.UserDisplayName: Shirish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T12:03:54.777Z
Answers.Text: (cont'd) When you say both $softmax(\hat u_i+c)$ and $softmax(\hat u_i+c)$ should both be equal to $e_s$ - do you mean $e_s^{1/2}$ (where this denotes a vector obtained on taking element-wise square root of $e_s$)? I'm guessing by $softmax(\hat u_i+c)*softmax(\hat u_i+c)$ you mean the vector obtained by element-wise multiplication of $softmax(\hat u_i+c)$ and $softmax(\hat u_i+c)$.
Answers.UserDisplayName: Shirish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T12:07:52.607Z
Answers.Text: Finally, I'm sorry I didn't quite understand your last paragraph. I guess by $\delta$ you mean $\mathcal{V}(e_i-e_j)$. I didn't understand from "and that all similar word pairs" onwards. Why do you say $v_a \approx v_b+const*\delta$? Is this $\delta$ different from the one in $\mathcal{U}\delta=k*1$? I don't see the reasoning behind the part "This constraint is heavy... ...would be low in practice". Sorry to be a pain but I want to understand all this properly, so I'd be really grateful if you could elaborate. Thanks again for the answer!
Answers.UserDisplayName: Shirish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T13:13:31.450Z
Answers.Text: @Shirish regarding your first 2 doubts: This is restriction isnt reality, its just an approximation to demonstrate the mathematics. We dont expect this to occur so plaoinly but as an expectation we expect it to be approximate
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T13:14:31.550Z
Answers.Text: @Shirish no i did mean $e_s$. Its multiplication, not addition: $e_s \odot e_s = e_s$
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T13:16:30.837Z
Answers.Text: @Shirish in essence all imilar words would have to have this connection linearly in the vector space, but because words have multiple senses and multiple meanings this is difficult for the optimization process to learn, if not impossible. Therefore it will probably do best in a scenario where the vectors magnitudes are not so different (i.e. $K=0, \delta=0$)
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T13:24:11.543Z
Answers.Text: Oh my bad about the $e_s$ doubt: stupid of me not to see that. Regarding your last comment - why $\delta=0$? Also, is the $\delta$ in $v_a\approx v_b+const*\delta$ the same as the one in $\mathcal{U}\delta=k*1$, or are they different? Did you mean to say that practically $v_a\approx v_b+\epsilon K$? Finally, I'm still not clear why the fact that this constraint is heavy, tough to learn and difficult to process would imply that $\epsilon$ is low. Sorry if I'm being dense.
Answers.UserDisplayName: Shirish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T13:28:22.833Z
Answers.Text: @Shirish No they are the same comment, one leads to the other and no $delta$ does not equal $\epsilon *K$ because $\delta = \mathcal{U}^{-1}K$, its heavy for various reasons: forces $K$ to be in image of $\mathcal{U}$ and $\delta$ to be in $\mathcal{V}$, along with this structure that would mean that all magnitudes would differ heavily which make the loss landscape difficult to traverse in a single order method like gd
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-16T13:43:04.157Z
Answers.Text: Could you edit the answer to show how one leads to the other (as in how we get $v_a\approx v_b+const*\delta$) and elaborate on the final paragraph in the answer? Just for future reference or if anyone else is to come across the same question. In any case, I've accepted the answer. Appreciate the help!
Answers.UserDisplayName: Shirish
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-15T14:36:11.270Z
Answers.LastActivityDate: 2019-08-16T13:48:25.033Z
Answers.LastEditDate: 2019-08-16T13:48:25.033Z
Answers.UserDisplayName: mshlis
Body: <p>This is related to <a href="https://ai.stackexchange.com/questions/13947/understanding-how-continuous-bag-of-words-method-learns-embedded-representations">my earlier question</a>, which I'm trying to break down into parts (this being the first). I'm reading notes on word vectors <a href="https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf" rel="nofollow noreferrer">here</a>. Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:</p>
  
  <p>If we have a vocabulary <span class="math-container">$V$</span>, the naive way to represent words in it would be via one-hot-encoding, or in other words, as basis vectors of <span class="math-container">$R^{|V|}$</span> - say <span class="math-container">$e_1, e_2,\ldots,e_{|V|}$</span>. We want to map these to <span class="math-container">$\mathbb{R}^n$</span>, via some linear transformation such that the images of similar words (more precisely, the images of basis vectors corresponding to similar words) have higher inner products. Assuming the matrix representation of the linear transformation given the standard basis of <span class="math-container">$\mathbb{R}^{|V|}$</span> is denoted by <span class="math-container">$\mathcal{V}$</span>, then the "embedding" of the <span class="math-container">$i$</span>-th vocab word (i.e. the image of the corresponding basis vector <span class="math-container">$e_i$</span> of <span class="math-container">$V$</span>) is given by <span class="math-container">$\mathcal{V}e_i$</span>.</p>
  
  <p>Now suppose we have a context "The cat ____ over a", CBoW seeks to find a word that would fit into this context. Let the words "the", "cat", "over", "a" be denoted (in the space <span class="math-container">$V$</span>) by <span class="math-container">$x_{i_1},x_{i_2},x_{i_3},x_{i_4}$</span> respectively. We take the image of their linear combination (in particular, their average):
  <span class="math-container">$$\hat v=\mathcal{V}\bigg(\frac{x_{i_1}+x_{i_2}+x_{i_3}+x_{i_4}}{4}\bigg)$$</span></p>
  
  <p>We then map <span class="math-container">$\hat v$</span> back from <span class="math-container">$\mathbb{R}^n$</span> to <span class="math-container">$\mathbb{R}^{|V|}$</span> via <em>another</em> linear mapping whose matrix representation is <span class="math-container">$\mathcal{U}$</span>: <span class="math-container">$$z=\mathcal{U}\hat v$$</span></p>
  
  <p>Then we turn this <em>score vector</em> <span class="math-container">$z$</span> into softmax probabilities <span class="math-container">$\hat y=softmax(z)$</span> and compare it to the basis vector corresponding to the actual word, say <span class="math-container">$e_c$</span>. For example, <span class="math-container">$e_c$</span> could be the basis vector corresponding to "jumped". </p>
  
  <p>Here's my interpretation of what this procedure is trying to do: given a context, we're trying to learn maps <span class="math-container">$\mathcal{U}$</span> and <span class="math-container">$\mathcal{V}$</span> such that given a context like "the cat ____ over a", the model should give a high score to words like "jumped" or "leaped", etc. Not just that - but "similar" contexts should also give rise to high scores for "jumped", "leaped", etc. For example, given a context "that dog ____ above this" wherein "that", "dog", "above", "this" are represented by <span class="math-container">$x_{j_1},x_{j_2},x_{j_3},x_{j_4}$</span>, let the image of their average be</p>
  
  <p><span class="math-container">$$\hat w=\mathcal{V}\bigg(\frac{x_{j_1}+x_{j_2}+x_{j_3}+x_{j_4}}{4}\bigg)$$</span></p>
  
  <p>This gets mapped to a score vector <span class="math-container">$z'=\mathcal{U}\hat w$</span>. Ideally, both score vectors <span class="math-container">$z$</span> and <span class="math-container">$z'$</span> should have similarly high magnitudes in their components corresponding to similar words "jumped" and "leaped".</p>
  
  <p>Is my above understanding correct? Consider the following quote from the lectures:</p>
  
  <blockquote>
    <p>We create two matrices, <span class="math-container">$\mathcal{V} \in \mathbb{R}^{n\times |V|}$</span> and <span class="math-container">$\mathcal{U} \in \mathbb{R}^{|V|\times n}$</span>, where <span class="math-container">$n$</span> is an arbitrary size which defines the size of our embedding space. <span class="math-container">$\mathcal{V}$</span> is the input word matrix such that the <span class="math-container">$i$</span>-th column of <span class="math-container">$\mathcal{V}$</span> is the <span class="math-container">$n$</span>-dimensional embedded vector for word <span class="math-container">$w_i$</span> when it is an input to this model. We denote this <span class="math-container">$n\times 1$</span> vector as <span class="math-container">$v_i$</span>. Similarly, <span class="math-container">$\mathcal{U}$</span> is the output word matrix. The <span class="math-container">$j$</span>-th row of <span class="math-container">$\mathcal{U}$</span> is an <span class="math-container">$n$</span>-dimensional embedded vector for word <span class="math-container">$w_j$</span> when it is an output of the model. We denote this row of <span class="math-container">$\mathcal{U}$</span> as <span class="math-container">$u_j$</span>.</p>
  </blockquote>
  
  <p><strong>It's not obvious to me why <span class="math-container">$v_i=\mathcal{V}e_i$</span> should be the same as or even similar to <span class="math-container">$u_i$</span>. How does the whole backpropagation procedure above ensure that?</strong></p>
  
  <p><strong>Also, how does the procedure ensure that basis vectors corresponding to similar words <span class="math-container">$e_i$</span> and <span class="math-container">$e_j$</span> are mapped to vectors in <span class="math-container">$\mathbb{R}^n$</span> that have high inner product?</strong> (In other words, how is it ensured that if words no. <span class="math-container">$i_1$</span> and <span class="math-container">$i_2$</span> are similar, then <span class="math-container">$\langle v_{i_1}, v_{i_2}\rangle$</span> and <span class="math-container">$\langle u_{i_1}, u_{i_2}\rangle$</span> have high values?)</p>
  
Tags: <natural-language-processing><word-embedding><word2vec>
Title: How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?
UserDisplayName: Shirish
ThreadText: # How does Continuous Bag of Words ensure that similar words are encoded as similar embeddings?
  
  (Asked by: username_0 on 2019-08-15)
  
  This is related to [my earlier question](https://ai.stackexchange.com/questions/13947/understanding-how-continuous-bag-of-words-method-learns-embedded-representations), which I'm trying to break down into parts (this being the first). I'm reading notes on word vectors [here](https://web.stanford.edu/class/cs224n/readings/cs224n-2019-notes01-wordvecs1.pdf). Specifically, I'm referring to section 4.2 on page 7. First, regarding points 1 to 6 - here's my understanding:
  
  If we have a vocabulary $V$, the naive way to represent words in it would be via one-hot-encoding, or in other words, as basis vectors of $R^{|V|}$ - say $e\_1, e\_2,\ldots,e\_{|V|}$. We want to map these to $\mathbb{R}^n$, via some linear transformation such that the images of similar words (more precisely, the images of basis vectors corresponding to similar words) have higher inner products. Assuming the matrix representation of the linear transformation given the standard basis of $\mathbb{R}^{|V|}$ is denoted by $\mathcal{V}$, then the "embedding" of the $i$-th vocab word (i.e. the image of the corresponding basis vector $e\_i$ of $V$) is given by $\mathcal{V}e\_i$.
  
  Now suppose we have a context "The cat \_\_\_\_ over a", CBoW seeks to find a word that would fit into this context. Let the words "the", "cat", "over", "a" be denoted (in the space $V$) by $x\_{i\_1},x\_{i\_2},x\_{i\_3},x\_{i\_4}$ respectively. We take the image of their linear combination (in particular, their average):
  $$\hat v=\mathcal{V}\bigg(\frac{x\_{i\_1}+x\_{i\_2}+x\_{i\_3}+x\_{i\_4}}{4}\bigg)$$
  
  We then map $\hat v$ back from $\mathbb{R}^n$ to $\mathbb{R}^{|V|}$ via *another* linear mapping whose matrix representation is $\mathcal{U}$: $$z=\mathcal{U}\hat v$$
  
  Then we turn this *score vector* $z$ into softmax probabilities $\hat y=softmax(z)$ and compare it to the basis vector corresponding to the actual word, say $e\_c$. For example, $e\_c$ could be the basis vector corresponding to "jumped".
  
  Here's my interpretation of what this procedure is trying to do: given a context, we're trying to learn maps $\mathcal{U}$ and $\mathcal{V}$ such that given a context like "the cat \_\_\_\_ over a", the model should give a high score to words like "jumped" or "leaped", etc. Not just that - but "similar" contexts should also give rise to high scores for "jumped", "leaped", etc. For example, given a context "that dog \_\_\_\_ above this" wherein "that", "dog", "above", "this" are represented by $x\_{j\_1},x\_{j\_2},x\_{j\_3},x\_{j\_4}$, let the image of their average be
  
  $$\hat w=\mathcal{V}\bigg(\frac{x\_{j\_1}+x\_{j\_2}+x\_{j\_3}+x\_{j\_4}}{4}\bigg)$$
  
  This gets mapped to a score vector $z'=\mathcal{U}\hat w$. Ideally, both score vectors $z$ and $z'$ should have similarly high magnitudes in their components corresponding to similar words "jumped" and "leaped".
  
  Is my above understanding correct? Consider the following quote from the lectures:
  
  > We create two matrices, $\mathcal{V} \in \mathbb{R}^{n\times |V|}$ and $\mathcal{U} \in \mathbb{R}^{|V|\times n}$, where $n$ is an arbitrary size which defines the size of our embedding space. $\mathcal{V}$ is the input word matrix such that the $i$-th column of $\mathcal{V}$ is the $n$-dimensional embedded vector for word $w\_i$ when it is an input to this model. We denote this $n\times 1$ vector as $v\_i$. Similarly, $\mathcal{U}$ is the output word matrix. The $j$-th row of $\mathcal{U}$ is an $n$-dimensional embedded vector for word $w\_j$ when it is an output of the model. We denote this row of $\mathcal{U}$ as $u\_j$.
  
  **It's not obvious to me why $v\_i=\mathcal{V}e\_i$ should be the same as or even similar to $u\_i$. How does the whole backpropagation procedure above ensure that?**
  
  **Also, how does the procedure ensure that basis vectors corresponding to similar words $e\_i$ and $e\_j$ are mapped to vectors in $\mathbb{R}^n$ that have high inner product?** (In other words, how is it ensured that if words no. $i\_1$ and $i\_2$ are similar, then $\langle v\_{i\_1}, v\_{i\_2}\rangle$ and $\langle u\_{i\_1}, u\_{i\_2}\rangle$ have high values?)
  
  **Tags:** natural-language-processing, word-embedding, word2vec
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-15. Score: 2 | Accepted answer
  
  Unlike in skip-gram, the reason similar words have similar embeddings in CBOW is because the words show up in the same contexts of other skipped words.
  
  lets assume two words $e\_i$ and $e\_j$ pop up in the exact same context of some word $e\_k$ with 3 other context words as well. An example would be:
  
  1. He *leaped* **over** the truck
  2. He *jumped* **over** the truck
  
  Where the italics represent the words with similar meanings but the bolded words above are the ones being predicted/skipped in CBOW. Lets now show this.
  
  Let the rest of the context be denoted $\{e\_r\}\_r$ and the skipped word as $e\_s$ and so the loss will try to minimize both $-e\_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e\_i + \sum\_re\_r)]))$ and $-e\_s^T log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e\_j + \sum\_re\_r)]))$.
  
  Assuming a long run withlarge enough batch-size (so ignoring catastrophic forgetting) we can essentially say it will be minimizing  
  $$-e\_s^T [log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e\_i + \sum\_re\_r)])) +\\ log(softmax(\mathcal{U}\mathcal{V}[\frac{1}{R+1}(e\_j + \sum\_re\_r)]))]$$  
  Now let $\mathcal{U}\mathcal{V}\frac{1}{R+1}\sum\_re\_r$ be denoted as $c$, $\mathcal{U}\mathcal{V}\frac{1}{R+1}e\_i$ as $\hat u\_i$, and $\mathcal{U}\mathcal{V}\frac{1}{R+1}e\_j$ as $\hat u\_j$. So we have to minimize  
  $$-e\_s^T[log(softmax(\hat u\_i+c)) + log(softmax(\hat u\_j+c))]$$  
  or equivalently  
  $$-e\_s^T\ log(softmax(\hat u\_i+c) \* softmax(\hat u\_j+c))$$  
  As we know in crossentropy, we get our critical point when the inside of the log here would equal $e\_s$, which is only possible if both $softmax(\hat u\_i+c)$ and $softmax(\hat u\_j+c)$ were to be equal to $e\_s$. softmax is non-invertible and is invariant under addition only, therefore to achieve this we would need $\hat u\_i + c$ to equal $\hat u\_j + c + K$ where $K$ is just some arbitrary constant vector ($K=[k,k,k,...]$. Subtracting $c$ and multiplying by $R+1$ we make our model want
  
  $$\begin{align}
  u\_i &= u\_j + K \\
  \implies K &= \mathcal{U}\mathcal{V}(e\_i -e\_j) \\
  \end{align}$$
  
  This would mean for all common words, we see this activity that $u\_{i-j}$ would be a constant, but this is difficult because that means a $\delta$ must exist such that $\mathcal{U}\delta = k\*\vec 1$ and that all similar word pairs indexed by $(a,b)$ would have $v\_a \approx v\_b + const\*\delta$ (because $u\_a = u\_b + K \implies v\_b = v\_a + \mathcal{U}^{-1}K$ and we denote $\delta = \mathcal{U}^{-1}K$). This constraint is heavy, tough to learn, and would also be difficult to process on multiple word senses that would appear in the vocabulary. This is because if it were to learn this, it enforces constraints on the images of $\mathcal{U}$ and $\mathcal{V}$ along with creating word vectors with highly different magnitudes making the learning process difficult. This would indicate that this constant would be low in practice. Therefore we would get $v\_i \approx v\_j$.
  
  > **Comments:**
  > **username_0**: Thanks for the nice answer! I have a few doubts: there's a couple of assumptions underlying the whole treatment - 1. The central (skipped) word as well as all **other** words in the context are **exactly** the same. 2. The definition of "similar" words assumed by your answer is ones that occur in the context of the **same** word (in this case, "over"). The definition is _not_: similar words are those that occur in same/similar contexts. Am I right that these are the assumptions? In their current form, aren't they quite restrictive/unlikely?  
  > **username_0**: (cont'd) When you say both $softmax(\hat u_i+c)$ and $softmax(\hat u_i+c)$ should both be equal to $e_s$ - do you mean $e_s^{1/2}$ (where this denotes a vector obtained on taking element-wise square root of $e_s$)? I'm guessing by $softmax(\hat u_i+c)*softmax(\hat u_i+c)$ you mean the vector obtained by element-wise multiplication of $softmax(\hat u_i+c)$ and $softmax(\hat u_i+c)$.  
  > **username_0**: Finally, I'm sorry I didn't quite understand your last paragraph. I guess by $\delta$ you mean $\mathcal{V}(e_i-e_j)$. I didn't understand from "and that all similar word pairs" onwards. Why do you say $v_a \approx v_b+const*\delta$? Is this $\delta$ different from the one in $\mathcal{U}\delta=k*1$? I don't see the reasoning behind the part "This constraint is heavy... ...would be low in practice". Sorry to be a pain but I want to understand all this properly, so I'd be really grateful if you could elaborate. Thanks again for the answer!  
  > **username_1**: @username_0 regarding your first 2 doubts: This is restriction isnt reality, its just an approximation to demonstrate the mathematics. We dont expect this to occur so plaoinly but as an expectation we expect it to be approximate  
  > **username_1**: @username_0 no i did mean $e_s$. Its multiplication, not addition: $e_s \odot e_s = e_s$  
  > **username_1**: @username_0 in essence all imilar words would have to have this connection linearly in the vector space, but because words have multiple senses and multiple meanings this is difficult for the optimization process to learn, if not impossible. Therefore it will probably do best in a scenario where the vectors magnitudes are not so different (i.e. $K=0, \delta=0$)  
  > **username_0**: Oh my bad about the $e_s$ doubt: stupid of me not to see that. Regarding your last comment - why $\delta=0$? Also, is the $\delta$ in $v_a\approx v_b+const*\delta$ the same as the one in $\mathcal{U}\delta=k*1$, or are they different? Did you mean to say that practically $v_a\approx v_b+\epsilon K$? Finally, I'm still not clear why the fact that this constraint is heavy, tough to learn and difficult to process would imply that $\epsilon$ is low. Sorry if I'm being dense.  
  > **username_1**: @username_0 No they are the same comment, one leads to the other and no $delta$ does not equal $\epsilon *K$ because $\delta = \mathcal{U}^{-1}K$, its heavy for various reasons: forces $K$ to be in image of $\mathcal{U}$ and $\delta$ to be in $\mathcal{V}$, along with this structure that would mean that all magnitudes would differ heavily which make the loss landscape difficult to traverse in a single order method like gd  
  > **username_0**: Could you edit the answer to show how one leads to the other (as in how we get $v_a\approx v_b+const*\delta$) and elaborate on the final paragraph in the answer? Just for future reference or if anyone else is to come across the same question. In any case, I've accepted the answer. Appreciate the help!  
  
  ---
  
  