Answers.Body: <p>Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value.</p>
  
  <p>For details: <a href="https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d#ecd3" rel="nofollow noreferrer">https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d#ecd3</a></p>
  
  <p>To feed data with different lengths, you can also try padding and trimming it. Set a constant L for length of paragraph and trim/pad all list of word vectors to this length. Padding adds 0 vectors to the begining of the list and trimming trims the first part of a text until it is equal to L. Even in LSTM networks padding and trimming is still used as even though you can feed as long of a text to a LSTM as you want, you still have to process the word vectors in batches, which requires them to be the same length.</p>
  
  <p>Example code in python on padding/trimming vector:</p>
  
  <pre><code>def pad_trim(list_vec, L):
      //list vec: {vec1,vec2,vec3......}
      //Assuming vecN have a size of 200
      return list_vec[-L:0] if len(list_vec) &gt; L else [[0] * 200] * (L-len(list_vec)) + list_vec
  </code></pre>
  
  <p>However in inference, you can ignore maximum length if you used a RNN based method, although as the network has not been trained on lengths more than L, it may perform better or worse.</p>
  
  <p>Generally speaking, you should go for concatenating if possible, so you can keep all information in the sentence. However both may work just fine depending on your task. </p>
  
  <p>For RNN based and CNN based model example, you should check this out:
  <a href="https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f" rel="nofollow noreferrer">https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-11T00:53:18.487Z
Answers.Text: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118298/discussion-on-answer-by-clement-hui-does-summing-up-word-vectors-destroy-their-m).
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-07T15:48:27.073Z
Answers.LastActivityDate: 2020-01-07T23:33:12.077Z
Answers.LastEditDate: 2020-01-07T23:33:12.077Z
Answers.UserDisplayName: Clement
Answers.Body: <blockquote>
    <p>But because the inputs have to have a fixed length</p>
  </blockquote>
  
  <p>Do they?  Why?  The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input sequentially and output the final classification of the paragraph.  This has the advantage of being able to take into account word order and constellations, as well as processing variable size inputs.</p>
  
  <p>Intuitively, I would think simply adding word vectors will include a lot of noise from commonly occurring words that don't provide much meaningful information from the classification.  I would consider Bayesian methods or dimensionality reduction methods to limit the input to the more useful input vectors.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T10:03:24.853Z
Answers.Text: How can I use an LSTM to input paragraphs with variable lengths to classify them?
Answers.UserDisplayName: Arnav Das
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-01-09T09:03:39.410Z
Answers.LastActivityDate: 2020-01-09T09:03:39.410Z
Answers.UserDisplayName: Ketil Malde
Body: <p>For example, I have a paragraph that I want to classify in a binary manner. But because the inputs have to have a fixed length, I need to ensure that every paragraph is represented by a uniform quantity.</p>
  <p>One thing I've done is taken every word in the paragraph, vectorized it using GloVe word2vec, and then summed up all of the vectors to create a &quot;paragraph&quot; vector, which I've then fed in as an input for my model. In doing so, have I destroyed any meaning the words might have possessed?</p>
  <p>Considering these two sentences would have the same vector:</p>
  <blockquote>
  <p>My dog bit Dave</p>
  </blockquote>
  <blockquote>
  <p>Dave bit my dog</p>
  </blockquote>
  <p>How do I get around this? Am I approaching this wrong?</p>
  <p>What other way can I train my model? If I take every word and feed that into my model, how do I know how many words I should take? How do I input these words? In the form of a 2D array, where each word vector is a column?</p>
  <p>I want to be able to train a model that can classify text accurately.
  Surprisingly, I'm getting a high (&gt;90%) for a relatively simple model like RandomForestClassifier just by using this summing up method. Any insights?</p>
  
Tags: <deep-learning><natural-language-processing><word-embedding><text-classification><binary-classification>
Title: Does summing up word vectors destroy their meaning?
UserDisplayName: Arnav Das
ThreadText: # Does summing up word vectors destroy their meaning?
  
  (Asked by: username_0 on 2020-01-07)
  
  For example, I have a paragraph that I want to classify in a binary manner. But because the inputs have to have a fixed length, I need to ensure that every paragraph is represented by a uniform quantity.
  
  One thing I've done is taken every word in the paragraph, vectorized it using GloVe word2vec, and then summed up all of the vectors to create a "paragraph" vector, which I've then fed in as an input for my model. In doing so, have I destroyed any meaning the words might have possessed?
  
  Considering these two sentences would have the same vector:
  
  > My dog bit Dave
  
  > Dave bit my dog
  
  How do I get around this? Am I approaching this wrong?
  
  What other way can I train my model? If I take every word and feed that into my model, how do I know how many words I should take? How do I input these words? In the form of a 2D array, where each word vector is a column?
  
  I want to be able to train a model that can classify text accurately.
  Surprisingly, I'm getting a high (>90%) for a relatively simple model like RandomForestClassifier just by using this summing up method. Any insights?
  
  **Tags:** deep-learning, natural-language-processing, word-embedding, text-classification, binary-classification
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-01-07. Score: 2
  
  Summing up a sequence of word vector maybe used in practice sometimes. However, the operation of addition is non-reversible, meaning that once you sum up a few numbers, you cannot get the original numbers back. However summing up a sequence of word vectors may work depending on your task. You should also normalize the values, or just use average value.
  For details: https://towardsdatascience.com/document-embedding-techniques-fed3e7a6a25d#ecd3
  To feed data with different lengths, you can also try padding and trimming it. Set a constant L for length of paragraph and trim/pad all list of word vectors to this length. Padding adds 0 vectors to the begining of the list and trimming trims the first part of a text until it is equal to L. Even in LSTM networks padding and trimming is still used as even though you can feed as long of a text to a LSTM as you want, you still have to process the word vectors in batches, which requires them to be the same length.
  Example code in python on padding/trimming vector:
  def pad_trim(list_vec, L):
      //list vec: {vec1,vec2,vec3......}
      //Assuming vecN have a size of 200
      return list_vec[-L:0] if len(list_vec) > L else [[0] * 200] * (L-len(list_vec)) + list_vec
  
  However in inference, you can ignore maximum length if you used a RNN based method, although as the network has not been trained on lengths more than L, it may perform better or worse.
  Generally speaking, you should go for concatenating if possible, so you can keep all information in the sentence. However both may work just fine depending on your task. 
  For RNN based and CNN based model example, you should check this out:
  https://medium.com/jatana/report-on-text-classification-using-cnn-rnn-han-f0e887214d5f
  
  > **Comments:**
  > **username_2**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/118298/discussion-on-answer-by-clement-hui-does-summing-up-word-vectors-destroy-their-m).  
  
  ---
  
  ### Answer by username_3 on 2020-01-09. Score: 1
  
  > But because the inputs have to have a fixed length
  
  Do they? Why? The go-to strategy would be to use an RNN (possibly with LSTM or GRUs, but probably not necessary) and train it to process input sequentially and output the final classification of the paragraph. This has the advantage of being able to take into account word order and constellations, as well as processing variable size inputs.
  
  Intuitively, I would think simply adding word vectors will include a lot of noise from commonly occurring words that don't provide much meaningful information from the classification. I would consider Bayesian methods or dimensionality reduction methods to limit the input to the more useful input vectors.
  
  > **Comments:**
  > **username_0**: How can I use an LSTM to input paragraphs with variable lengths to classify them?  
  
  ---
  
  