Answers.Body: <p>Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words.</p>
  <p><a href="https://i.sstatic.net/2VUM3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/2VUM3.png" alt="enter image description here" /></a></p>
  <p>You could define it like this:</p>
  <pre><code>num_of_words = 50000
  # one hot encoded word
  input = Input(num_of_words)
  # You could use non linear activation
  w2v = Dense(300, activation=”linear”)(input)
  output = Dense(num_of_words, activation=”softmax”)(w2v)
  </code></pre>
  <p>But in practice, the model is redefined and takes two words as input and predicts the next words. It outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).</p>
  <p><a href="https://i.sstatic.net/9h4VT.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/9h4VT.png" alt="enter image description here" /></a></p>
  <p>It is trained both ways from the beginning to the end of a sentence and in the reverse direction. The loss used is categorical_crossentropy. A detailed explanation can be found here [http://jalammar.github.io/illustrated-word2vec/]</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-12T13:13:34.387Z
Answers.Text: thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network?
Answers.UserDisplayName: JumpJehovah
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-12T13:50:59.547Z
Answers.Text: It depends what you want to achieve And what kind of hardware you have. I trained  it for 1 million words on a regular laptop with gpu.  I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets.
Answers.UserDisplayName: user301845
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-09T17:16:13.710Z
Answers.LastActivityDate: 2022-08-01T16:18:05.537Z
Answers.LastEditDate: 2022-08-01T16:18:05.537Z
Answers.UserDisplayName: user301845
Body: <p>For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.</p>
  <p>Is it much different from a classic network, with weights and biases? How does it figure out the loss?</p>
  <p>If someone could point me towards a guide to how these things work exactly I would be very grateful.</p>
  
Tags: <neural-networks><implementation><word-embedding>
Title: How can I create an embedding layer to convert words to a vector space from scratch?
UserDisplayName: JumpJehovah
ThreadText: # How can I create an embedding layer to convert words to a vector space from scratch?
  
  (Asked by: username_0 on 2020-07-09)
  
  For an upcoming project, I am trying to build a neural network for classifying text from scratch, without the use of libraries. This requires an embedding layer, or a way to convert words to some vector representation. I understand the gist, but I can't find any deep explanations or tutorials that don't start with importing TensorFlow. All I'm really told is that it works by context using a few surrounding words, but I don't understand exactly how.
  
  Is it much different from a classic network, with weights and biases? How does it figure out the loss?
  
  If someone could point me towards a guide to how these things work exactly I would be very grateful.
  
  **Tags:** neural-networks, implementation, word-embedding
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-09. Score: 1
  
  Word2vec embeddings are trained using a simple auto-encoder model that takes a word and tries to predict one word from the window of surrounding words.
  
  [![enter image description here](https://i.sstatic.net/2VUM3.png)](https://i.sstatic.net/2VUM3.png)
  
  You could define it like this:
  
  ```
  num_of_words = 50000
  # one hot encoded word
  input = Input(num_of_words)
  # You could use non linear activation
  w2v = Dense(300, activation=”linear”)(input)
  output = Dense(num_of_words, activation=”softmax”)(w2v)
  
  ```
  
  But in practice, the model is redefined and takes two words as input and predicts the next words. It outputs a probability score for all the words it knows (the model’s “vocabulary”, which can range from a few thousand to over a million words).
  
  [![enter image description here](https://i.sstatic.net/9h4VT.png)](https://i.sstatic.net/9h4VT.png)
  
  It is trained both ways from the beginning to the end of a sentence and in the reverse direction. The loss used is categorical\_crossentropy. A detailed explanation can be found here [http://jalammar.github.io/illustrated-word2vec/]
  
  > **Comments:**
  > **username_0**: thank you for this, ill read the article you have mentioned, but if I could also ask how many words is a good amount to use to train such a network?  
  > **username_1**: It depends what you want to achieve And what kind of hardware you have. I trained  it for 1 million words on a regular laptop with gpu.  I did not want to reinvent the wheel so I have used gensim. If There is a pretrained model for your task/language then I would go for it as it usually is trained on bigger datasets.  
  
  ---
  
  