Answers.Body: <p>There are multiple ways to get word embedding from a corpus.</p>
  <ul>
  <li><strong>Count Vectorizer:</strong> You can use the <code>CountVectorizer()</code> from <code>sklearn.feature_extraction.text</code> and then use the <code>fit_transform()</code> if the corpus has been converted into a list of sentences</li>
  <li><strong>TF-IDF Vectorizer:</strong> You can use the <code>TfidfVectorizer</code> from <code>sklearn.feature_extraction.text</code> and then again use the <code>fit_transform()</code> on a list of sentences</li>
  <li><strong>word2vec:</strong> You can make a <code>word2vec</code> model from <code>gensim.models</code> by using  <code>word2vec.Word2vec</code>.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T01:40:45.277Z
Answers.LastActivityDate: 2020-08-25T01:40:45.277Z
Answers.UserDisplayName: Saurav Maheshkar
Answers.Body: <p>Assuming that the dictionary of the words, that your model comes up with, is a subset of the pretrained embeddings, for example of Google's pretrained word2vec, then it is maybe a better option following these embeddings, if your model can handle that size of dimension.</p>
  <p>However, sometimes that would not always be the best solution, taking into account the nature of the problem. For example, if you are trying to use NLP on medical texts that contain rare and special words, then maybe you should use your embedding layer, assuming that you have an adequate data size, or both of them. That is just a thought of mine. For sure, there can be several other use cases which should propose the embedding layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-24T21:29:49.587Z
Answers.LastActivityDate: 2020-08-24T21:29:49.587Z
Answers.UserDisplayName: ddaedalus
Answers.Body: <p>If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part of the graph.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:51:11.373Z
Answers.Text: I very recently came to the same conclusion. There is also another benefit: I can train in larger batches
Answers.UserDisplayName: chessprogrammer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-08-25T13:36:46.010Z
Answers.LastActivityDate: 2020-08-25T13:36:46.010Z
Answers.UserDisplayName: Andy
Body: <p>When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings.</p>
  <p>However, assuming the embeddings are pre-trained and frozen, there is another option. We could simply preprocess the training data prior to giving it to the model so that it is already converted to the embeddings. This will speed up training, since this conversion need only be performed once, as opposed to on the fly for each epoch.</p>
  <p>Thus, the second option seems better. But the first choice seems more common. Assuming the embeddings are pre-trained and frozen, is there a reason I might choose the first option over the second?</p>
  
Tags: <natural-language-processing><data-preprocessing><word-embedding>
Title: When to convert data to word embeddings in NLP
UserDisplayName: chessprogrammer
ThreadText: # When to convert data to word embeddings in NLP
  
  (Asked by: username_0 on 2020-08-21)
  
  When training a network using word embeddings, it is standard to add an embedding layer to first convert the input vector to the embeddings.
  
  However, assuming the embeddings are pre-trained and frozen, there is another option. We could simply preprocess the training data prior to giving it to the model so that it is already converted to the embeddings. This will speed up training, since this conversion need only be performed once, as opposed to on the fly for each epoch.
  
  Thus, the second option seems better. But the first choice seems more common. Assuming the embeddings are pre-trained and frozen, is there a reason I might choose the first option over the second?
  
  **Tags:** natural-language-processing, data-preprocessing, word-embedding
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_3 on 2020-08-25. Score: 2 | Accepted answer
  
  If you have to move a lot of data around during training (like retrieving batches from disk/network/what have you), it's much faster to do so as a rank-3 tensor of [batches, documents, indices] than as a rank-4 tensor of [batches, documents, indices, vectors]. In this case, while the embedding is O(1) wherever you put it, it's more efficient to do so as part of the graph.
  
  > **Comments:**
  > **username_0**: I very recently came to the same conclusion. There is also another benefit: I can train in larger batches  
  
  ---
  
  ### Answer by username_1 on 2020-08-25. Score: 1
  
  There are multiple ways to get word embedding from a corpus.
  
  * **Count Vectorizer:** You can use the `CountVectorizer()` from `sklearn.feature_extraction.text` and then use the `fit_transform()` if the corpus has been converted into a list of sentences
  * **TF-IDF Vectorizer:** You can use the `TfidfVectorizer` from `sklearn.feature_extraction.text` and then again use the `fit_transform()` on a list of sentences
  * **word2vec:** You can make a `word2vec` model from `gensim.models` by using `word2vec.Word2vec`.
  
  
  ---
  
  ### Answer by username_2 on 2020-08-24. Score: 0
  
  Assuming that the dictionary of the words, that your model comes up with, is a subset of the pretrained embeddings, for example of Google's pretrained word2vec, then it is maybe a better option following these embeddings, if your model can handle that size of dimension.
  
  However, sometimes that would not always be the best solution, taking into account the nature of the problem. For example, if you are trying to use NLP on medical texts that contain rare and special words, then maybe you should use your embedding layer, assuming that you have an adequate data size, or both of them. That is just a thought of mine. For sure, there can be several other use cases which should propose the embedding layer.
  
  
  ---
  
  