Answers.Body: <p>The typical example used to explain word embeddings to beginners is to conceptualize each word embedding as a vector. When a word embedder is successfully trained, it should enable operations such as:
  <span class="math-container">$$
  \text{king} - \text{man} + \text{woman} = \text{queen}
  $$</span>
  Let each embedding be denoted as <span class="math-container">$\vec{x}_{t}$</span>, where <span class="math-container">$t$</span> represents a word. . In practice, the following quantity is computed:
  <span class="math-container">$$
  \vec{y} = \vec{x}_{\text{king}} - \vec{x}_{\text{man}} + \vec{x}_{\text{woman}}
  $$</span>
  Next, one searches for the closest embedding <span class="math-container">$\vec{x}_{t}$</span> to <span class="math-container">$\vec{y}$</span>. In this case, <span class="math-container">$\vec{x}_{\text{queen}}$</span> is expected to be the closest embedding to <span class="math-container">$\vec{y}$</span>.</p>
  <p>Thus, to answer your question: yes, this has been implemented in studies such as <a href="https://aclanthology.org/N13-1090.pdf" rel="nofollow noreferrer">&quot;Linguistic Regularities in Continuous Space Word Representations&quot; by Mikolov et al.</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-10T16:32:43.757Z
Answers.LastActivityDate: 2024-07-10T16:32:43.757Z
Answers.UserDisplayName: frad
Body: <p>In the transformer architecture, one of the step is to update the embeddings, allowing words to pass information to whichever words they are relevant to. For example in the sentence &quot;a small cat is walking in the garden&quot; we want the embedding of &quot;small&quot; to cause a change to &quot;cat&quot; and to moves it to a different part of the embedding space that more specifically encodes a &quot;small cat&quot;. &quot;small&quot; is an adjective but the same thing can be done with verbs, pronouns, ....</p>
  <p>If we would take the updated embeddings and start looking for the closest tokens (the ones that minimize distance), what would we got ? Are the updated embeddings a vector that only make sense inside the transformer layers and for prediction (the last step), or do they somehow still have a relation with the initial embeddings used for tokens ?</p>
  <p>Is this plausible that the updated embedding of &quot;small cat&quot; to be closer to &quot;kitty&quot; than &quot;cat&quot; ?</p>
  
Tags: <transformer><large-language-models>
Title: Would converting embeddings updated by a transformer into tokens (eg: by searching for the nearest embeddings) produce results that make sense?
UserDisplayName: tigrou
ThreadText: # Would converting embeddings updated by a transformer into tokens (eg: by searching for the nearest embeddings) produce results that make sense?
  
  (Asked by: username_0 on 2024-06-26)
  
  In the transformer architecture, one of the step is to update the embeddings, allowing words to pass information to whichever words they are relevant to. For example in the sentence "a small cat is walking in the garden" we want the embedding of "small" to cause a change to "cat" and to moves it to a different part of the embedding space that more specifically encodes a "small cat". "small" is an adjective but the same thing can be done with verbs, pronouns, ....
  
  If we would take the updated embeddings and start looking for the closest tokens (the ones that minimize distance), what would we got ? Are the updated embeddings a vector that only make sense inside the transformer layers and for prediction (the last step), or do they somehow still have a relation with the initial embeddings used for tokens ?
  
  Is this plausible that the updated embedding of "small cat" to be closer to "kitty" than "cat" ?
  
  **Tags:** transformer, large-language-models
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-07-10. Score: 1
  
  The typical example used to explain word embeddings to beginners is to conceptualize each word embedding as a vector. When a word embedder is successfully trained, it should enable operations such as:
  $$
  \text{king} - \text{man} + \text{woman} = \text{queen}
  $$
  Let each embedding be denoted as $\vec{x}\_{t}$, where $t$ represents a word. . In practice, the following quantity is computed:
  $$
  \vec{y} = \vec{x}\_{\text{king}} - \vec{x}\_{\text{man}} + \vec{x}\_{\text{woman}}
  $$
  Next, one searches for the closest embedding $\vec{x}\_{t}$ to $\vec{y}$. In this case, $\vec{x}\_{\text{queen}}$ is expected to be the closest embedding to $\vec{y}$.
  
  Thus, to answer your question: yes, this has been implemented in studies such as ["Linguistic Regularities in Continuous Space Word Representations" by Mikolov et al.](https://aclanthology.org/N13-1090.pdf)
  
  
  ---
  
  