Answers.Body: <p>Neither of these approaches really represent the word order, though it could be argued that a bag of n-grams has <em>some</em> word order representation in it.</p>
  <p>Remember that, using bag of words, each word is assigned a value and then the aggregate values are converted into a multi-hot encoded row vector.  In other words, a sentence like:</p>
  <blockquote>
  <p>This is a test.</p>
  </blockquote>
  <p>Would likely be preprocessed to:</p>
  <blockquote>
  <p>this is a test</p>
  </blockquote>
  <p>And then converted to a numeric representation:</p>
  <p><code>[15, 12, 3, 56]</code></p>
  <p>where each of the numbers is the number used to encode or tokenize that word.  The next step would be to convert this to a multi-hot encoded vector:</p>
  <p><code>[ 0, 0, 0, 0, ..., 1, 0, 0, 1, 0, 0, ...]</code></p>
  <p>where the length of the vector is the length of the number of words in the vocabulary.</p>
  <p>As a result of this representation, all information about word order is lost.</p>
  <p>Now consider the case of bag of n-grams.  Let's assume we used an <span class="math-container">$n=3$</span> for the number of n-grams.  This means that, rather than tokenizing each word individually, we are now converting each trio of words into a token.  These tokens are then, again, converted to a multi-hot encoded vector.</p>
  <p>While it is certainly true that there is <em>some</em> word order information retained within the n-grams themselves, the order of the n-grams in the text is still lost when we convert to the multi-hot encoded vector.</p>
  <p>TLDR:</p>
  <p>Bag of words preserves no word order at all, so the computational efficiency with regard to that is non-existent.  Bag of n-grams preserves <em>some</em> word order, but not much.  Bag of n-grams is certainly an easy thing to implement, especially with variable length inputs, so in that sense it might be more efficient to use over other approaches, such as a transformer, LSTM, or an embedding layer with a Conv1D.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T16:04:19.043Z
Answers.Text: brilliant! Any recommended source where I can get a broader view on this topic (bag of n-grams vs BOW)?
Answers.UserDisplayName: Filippo Colombi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T23:40:50.270Z
Answers.Text: Have you tried the deep learning book? https://www.deeplearningbook.org/
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T09:44:57.870Z
Answers.Text: appreciate that!
Answers.UserDisplayName: Filippo Colombi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T11:34:19.863Z
Answers.Text: If this answer works for you, consider upvoting and accepting it, please. @FilippoColombi
Answers.UserDisplayName: David Hoelzer
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T15:23:27.060Z
Answers.LastActivityDate: 2023-02-02T15:23:27.060Z
Answers.UserDisplayName: David Hoelzer
Body: <p>I cannot figure out what is more computationally efficient between the two representations mentioned in my question in terms of training time and the amount of data required. Especially, when it comes to considering the order of the words in a sentence.</p>
  
Tags: <machine-learning><natural-language-processing><word-embedding>
Title: What's computationally more efficient between bag-of-words representation and bag-of-ngrams representation, with special regard to words order?
UserDisplayName: Filippo Colombi
ThreadText: # What's computationally more efficient between bag-of-words representation and bag-of-ngrams representation, with special regard to words order?
  
  (Asked by: username_0 on 2023-02-02)
  
  I cannot figure out what is more computationally efficient between the two representations mentioned in my question in terms of training time and the amount of data required. Especially, when it comes to considering the order of the words in a sentence.
  
  **Tags:** machine-learning, natural-language-processing, word-embedding
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-02-02. Score: 1 | Accepted answer
  
  Neither of these approaches really represent the word order, though it could be argued that a bag of n-grams has *some* word order representation in it.
  
  Remember that, using bag of words, each word is assigned a value and then the aggregate values are converted into a multi-hot encoded row vector. In other words, a sentence like:
  
  > This is a test.
  
  Would likely be preprocessed to:
  
  > this is a test
  
  And then converted to a numeric representation:
  
  `[15, 12, 3, 56]`
  
  where each of the numbers is the number used to encode or tokenize that word. The next step would be to convert this to a multi-hot encoded vector:
  
  `[ 0, 0, 0, 0, ..., 1, 0, 0, 1, 0, 0, ...]`
  
  where the length of the vector is the length of the number of words in the vocabulary.
  
  As a result of this representation, all information about word order is lost.
  
  Now consider the case of bag of n-grams. Let's assume we used an $n=3$ for the number of n-grams. This means that, rather than tokenizing each word individually, we are now converting each trio of words into a token. These tokens are then, again, converted to a multi-hot encoded vector.
  
  While it is certainly true that there is *some* word order information retained within the n-grams themselves, the order of the n-grams in the text is still lost when we convert to the multi-hot encoded vector.
  
  TLDR:
  
  Bag of words preserves no word order at all, so the computational efficiency with regard to that is non-existent. Bag of n-grams preserves *some* word order, but not much. Bag of n-grams is certainly an easy thing to implement, especially with variable length inputs, so in that sense it might be more efficient to use over other approaches, such as a transformer, LSTM, or an embedding layer with a Conv1D.
  
  > **Comments:**
  > **username_0**: brilliant! Any recommended source where I can get a broader view on this topic (bag of n-grams vs BOW)?  
  > **username_1**: Have you tried the deep learning book? https://www.deeplearningbook.org/  
  > **username_0**: appreciate that!  
  > **username_1**: If this answer works for you, consider upvoting and accepting it, please. @username_0  
  
  ---
  
  