Answers.Body: <p>Whether linear maps are &quot;required&quot; is an extremely difficult question -- I'm sure there's a number of ways to implement the general idea for the skip gram architecture. But this is more of a research question.</p>
  <p>This might actually be the intention of your question, but there are definitely clear reasons why linear maps are an <em>intuitive</em> choice.</p>
  <p>As discussed in the post, the goal is the convert the sparse one-hot representations into dense vectors. <a href="https://web.stanford.edu/class/cs224n/slides/cs224n-2024-lecture01-wordvecs1-public.pdf" rel="nofollow noreferrer">One intuition is that the meaning of words is determined by it's neighbors</a> -- so skip-gram learns a model that can project one-hot representations into a lower dimensional representation <strong>such that that representation can be used to predict the held out word</strong>. This may remind you of autoencoders. Finally, linear maps are an extremely easy way to project vectors into different dimensions (it's just a matrix multiply!), making it an intuitive choice here.</p>
  <p>In fact, reading the <a href="https://arxiv.org/pdf/1301.3781.pdf" rel="nofollow noreferrer">original paper</a>, the mention:</p>
  <blockquote>
  <p>The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model. While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.</p>
  </blockquote>
  <p>i.e., linear maps are appealing because they're simple.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-19T00:36:50.403Z
Answers.LastActivityDate: 2024-01-19T00:36:50.403Z
Answers.UserDisplayName: Alexander Wan
Body: <p>This question is inspired from Lilian Weng's blog <a href="https://lilianweng.github.io/posts/2017-10-15-word-embedding/#other-tips-for-learning-word-embedding" rel="nofollow noreferrer">here</a> about skip-gram model, where she shows the model as multiplication via 2 matrices <span class="math-container">$W$</span> and <span class="math-container">$W'$</span> for embedding and word context matrix and <span class="math-container">$W$</span> takes in one hot encoded embedding into a 'better' embedding space.</p>
  <p>My question is, why should we require a linear map for this task? As in, why should some linear map from one-hot encoded space be useful for embedding words densely, while preserving semantic information as much as possible (this is the definition of word 'better' as used above)?</p>
  <p>For an overview of the model, see here:</p>
  <p><a href="https://i.sstatic.net/9B6bb.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9B6bb.png" alt="enter image description here" /></a></p>
  
Tags: <natural-language-processing><word-embedding><skip-gram>
Title: Why does skip-gram uses linear maps for embedding?
UserDisplayName: Mahammad Yusifov
ThreadText: # Why does skip-gram uses linear maps for embedding?
  
  (Asked by: username_0 on 2024-01-18)
  
  This question is inspired from Lilian Weng's blog [here](https://lilianweng.github.io/posts/2017-10-15-word-embedding/#other-tips-for-learning-word-embedding) about skip-gram model, where she shows the model as multiplication via 2 matrices $W$ and $W'$ for embedding and word context matrix and $W$ takes in one hot encoded embedding into a 'better' embedding space.
  
  My question is, why should we require a linear map for this task? As in, why should some linear map from one-hot encoded space be useful for embedding words densely, while preserving semantic information as much as possible (this is the definition of word 'better' as used above)?
  
  For an overview of the model, see here:
  
  [![enter image description here](https://i.sstatic.net/9B6bb.png)](https://i.sstatic.net/9B6bb.png)
  
  **Tags:** natural-language-processing, word-embedding, skip-gram
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-01-19. Score: 1 | Accepted answer
  
  Whether linear maps are "required" is an extremely difficult question -- I'm sure there's a number of ways to implement the general idea for the skip gram architecture. But this is more of a research question.
  
  This might actually be the intention of your question, but there are definitely clear reasons why linear maps are an *intuitive* choice.
  
  As discussed in the post, the goal is the convert the sparse one-hot representations into dense vectors. [One intuition is that the meaning of words is determined by it's neighbors](https://web.stanford.edu/class/cs224n/slides/cs224n-2024-lecture01-wordvecs1-public.pdf) -- so skip-gram learns a model that can project one-hot representations into a lower dimensional representation **such that that representation can be used to predict the held out word**. This may remind you of autoencoders. Finally, linear maps are an extremely easy way to project vectors into different dimensions (it's just a matrix multiply!), making it an intuitive choice here.
  
  In fact, reading the [original paper](https://arxiv.org/pdf/1301.3781.pdf), the mention:
  
  > The main observation from the previous section was that most of the complexity is caused by the non-linear hidden layer in the model. While this is what makes neural networks so attractive, we decided to explore simpler models that might not be able to represent the data as precisely as neural networks, but can possibly be trained on much more data efficiently.
  
  i.e., linear maps are appealing because they're simple.
  
  
  ---
  
  