<p>Actually, LSTM is not used to get word2vec. Indeed, word2vec is extracted from corpus of words using MLP (Multi Layer Perceptron). There is a great tutorial on how to extract word2wec:
<a href="http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/" rel="nofollow noreferrer">http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/</a></p>

<p>After representing word as vectors, you feed your text to LSTM in a deep architecture which the last layer must be softmax to categorize your text.</p>

CC BY-SA 4.0
2018-05-30T18:06:32.227Z
2018-05-30T18:06:32.227Z
pythinker
<p>In the research papers, it is not clear how they do that. From what I understood, you need to add a dense layer after your RNN layer. This dense layer is the size of your vocabulary. From my experience, this works even for a large vocabulary (30 000 - 40 000 for me) if you have enough data. Here you don't try to reconstruct the embedding but a one-hot vector of the current word. You can then use a cross-entropy loss. This last layer will have a lot of parameters.</p>

<p>You will see several implementations which are using the MSE loss directly on the word embedding output. Personally I didn't succeed with this approach but if other people could share their experiences, it could be great. </p>

CC BY-SA 4.0
2018-05-30T12:46:58.630Z
2018-05-30T12:46:58.630Z
Adrien D
<p>I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot vectors as long as the vocabulary (which is too long). So, should I use a linear activation function on the output to get the word embeddings directly (and then find the closest word) or is there something I'm missing here?</p>

<natural-language-processing><long-short-term-memory><word-embedding><word2vec>
How should the output layer of an LSTM be when the output are word embeddings?
George Sechu
# How should the output layer of an LSTM be when the output are word embeddings?

(Asked by: username_0 on 2018-05-30)

I'm having trouble grasping how to output word embeddings from an LSTM model. I'm seeing many examples using a softmax activation function on the output, but for that I would need to output one hot vectors as long as the vocabulary (which is too long). So, should I use a linear activation function on the output to get the word embeddings directly (and then find the closest word) or is there something I'm missing here?

**Tags:** natural-language-processing, long-short-term-memory, word-embedding, word2vec


---

## 2 answers

### Answer by username_1 on 2018-05-30. Score: 1

Actually, LSTM is not used to get word2vec. Indeed, word2vec is extracted from corpus of words using MLP (Multi Layer Perceptron). There is a great tutorial on how to extract word2wec:
<http://mccormickml.com/2016/04/19/word2vec-tutorial-the-skip-gram-model/>

After representing word as vectors, you feed your text to LSTM in a deep architecture which the last layer must be softmax to categorize your text.


---

### Answer by username_2 on 2018-05-30. Score: 0

In the research papers, it is not clear how they do that. From what I understood, you need to add a dense layer after your RNN layer. This dense layer is the size of your vocabulary. From my experience, this works even for a large vocabulary (30 000 - 40 000 for me) if you have enough data. Here you don't try to reconstruct the embedding but a one-hot vector of the current word. You can then use a cross-entropy loss. This last layer will have a lot of parameters.

You will see several implementations which are using the MSE loss directly on the word embedding output. Personally I didn't succeed with this approach but if other people could share their experiences, it could be great.


---

