Answers.Body: <p>Actually, LSTM is not used to get word2vec. Indeed, word2vec is extracted from corpus of words using MLP (Multi Layer Perceptron). There is a great tutorial on how to extract word2wec:
  <a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/" rel="nofollow noreferrer">http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/</a></p>
  
  <p>After representing word as vectors, you feed your text to LSTM in a deep architecture which the last layer must be softmax to categorize your text.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-30T18:06:32.227Z
Answers.LastActivityDate: 2018-05-30T18:06:32.227Z
Answers.UserDisplayName: pythinker
Answers.Body: <p>In the research papers, it is not clear how they do that. From what I understood, you need to add a dense layer after your RNN layer. This dense layer is the size of your vocabulary. From my experience, this works even for a large vocabulary (30 000 - 40 000 for me) if you have enough data. Here you don't try to reconstruct the embedding but a one-hot vector of the current word. You can then use a cross-entropy loss. This last layer will have a lot of parameters.</p>
  
  <p>You will see several implementations which are using the MSE loss directly on the word embedding output. Personally I didn't succeed with this approach but if other people could share their experiences, it could be great. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-30T12:46:58.630Z
Answers.LastActivityDate: 2018-05-30T12:46:58.630Z
Answers.UserDisplayName: Adrien D
Body: <p>I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot vectors as long as the vocabulary (which is too long). So, should I use a linear activation function on the output to get the word embeddings directly (and then find the closest word) or is there something I'm missing here?</p>
  
Tags: <natural-language-processing><long-short-term-memory><word-embedding><word2vec>
Title: How should the output layer of an LSTM be when the output are word embeddings?
UserDisplayName: George Sechu
ThreadText: # How should the output layer of an LSTM be when the output are word embeddings?
  
  (Asked by: username_0 on 2018-05-30)
  
  I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot vectors as long as the vocabulary (which is too long). So, should I use a linear activation function on the output to get the word embeddings directly (and then find the closest word) or is there something I'm missing here?
  
  **Tags:** natural-language-processing, long-short-term-memory, word-embedding, word2vec
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-05-30. Score: 1
  
  Actually, LSTM is not used to get word2vec. Indeed, word2vec is extracted from corpus of words using MLP (Multi Layer Perceptron). There is a great tutorial on how to extract word2wec:
  <http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/>
  
  After representing word as vectors, you feed your text to LSTM in a deep architecture which the last layer must be softmax to categorize your text.
  
  
  ---
  
  ### Answer by username_2 on 2018-05-30. Score: 0
  
  In the research papers, it is not clear how they do that. From what I understood, you need to add a dense layer after your RNN layer. This dense layer is the size of your vocabulary. From my experience, this works even for a large vocabulary (30 000 - 40 000 for me) if you have enough data. Here you don't try to reconstruct the embedding but a one-hot vector of the current word. You can then use a cross-entropy loss. This last layer will have a lot of parameters.
  
  You will see several implementations which are using the MSE loss directly on the word embedding output. Personally I didn't succeed with this approach but if other people could share their experiences, it could be great.
  
  
  ---
  
  