<p>Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words.</p>
<p><a href="https://i.sstatic.net/2VUM3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2VUM3.png" alt="enter image description here" /></a></p>
<p>You could define it like this:</p>
<pre><code>num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)
</code></pre>
<p>But in practice, the model is redefined and takes two words as input and predicts the next words. It outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).</p>
<p><a href="https://i.sstatic.net/9h4VT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9h4VT.png" alt="enter image description here" /></a></p>
<p>It is trained both ways from the beginning to the end of a sentence and in the reverse direction. The loss used is categorical_crossentropy. A detailed explanation can be found here [http://jalammar.github.io/illustrated-word2vec/]</p>

CC BY-SA 4.0
2020-07-12T13:13:34.387Z
thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network?
JumpJehovah
CC BY-SA 4.0
2020-07-12T13:50:59.547Z
It depends what you want to achieve And what kind of hardware you have. I trained  it for 1 million words on a regular laptop with gpu.  I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets.
user301845
CC BY-SA 4.0
2020-07-09T17:16:13.710Z
2022-08-01T16:18:05.537Z
2022-08-01T16:18:05.537Z
user301845
<p>For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.</p>
<p>Is it much different from a classic network, with weights and biases? How does it figure out the loss?</p>
<p>If someone could point me towards a guide to how these things work exactly I would be very grateful.</p>

<neural-networks><implementation><word-embedding>
How can I create an embedding layer to convert words to a vector space from scratch?
JumpJehovah
# How can I create an embedding layer to convert words to a vector space from scratch?

(Asked by: username_0 on 2020-07-09)

For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.

Is it much different from a classic network, with weights and biases? How does it figure out the loss?

If someone could point me towards a guide to how these things work exactly I would be very grateful.

**Tags:** neural-networks, implementation, word-embedding


---

## 1 answers

### Answer by username_1 on 2020-07-09. Score: 1

Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words.

You could define it like this:
num_of_words = 50000
# one hot encoded word
input = Input(num_of_words)
# You could use non linear activation
w2v = Dense(300, activation=”linear”)(input)
output = Dense(num_of_words, activation=”softmax”)(w2v)

But in practice, the model is redefined and takes two words as input and predicts the next words. It outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).

It is trained both ways from the beginning to the end of a sentence and in the reverse direction. The loss used is categorical_crossentropy. A detailed explanation can be found here [http://jalammar.github.io/illustrated-word2vec/]

> **Comments:**
> **username_0**: thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network?  
> **username_1**: It depends what you want to achieve And what kind of hardware you have. I trained  it for 1 million words on a regular laptop with gpu.  I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets.  

---

