Answers.Body: <p>Yes, N-grams is about <em>joining</em> <span class="math-container">$n$</span> words as one single token.</p>
  <p>Keep in mind it will greatly increase your features size:</p>
  <p>If you originally have 1000 unique words, notice you could get up to 1000² 2-gram (usually you don't get ALL the combinations, but notice the number of features can potentially grow huge!)</p>
  <p>If your dataset contains between thousands or millions samples, it could be enough to train a simple bag-of-words. But when you use a bi-gram, you'll probably need at least a million samples and lot more training steps. Besides, you'll probably have to tweak the hyperparameters.</p>
  <p>That's a common Machine Learning trade-off. A simple model is not very accurate. But a more complex model requires more data, more training and can overfit more easily.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T12:20:47.343Z
Answers.Text: It seems I am constantly facing a lack of data to train the kind of models I'd like to learn.
Answers.UserDisplayName: rocksNwaves
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-30T12:24:22.843Z
Answers.Text: Welcome to machine learning!
Answers.UserDisplayName: Andre Goulart
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-28T03:46:26.170Z
Answers.LastActivityDate: 2021-08-28T03:46:26.170Z
Answers.UserDisplayName: Andre Goulart
Body: <p>I've been playing with PyTorch's <code>nn.EmbeddingBag</code> for sentence classification for about a month. I've been doing some feature engineering, playing with different tokenizers, etc. I'm just trying to get the best performance out of this simple model as I can. I'm new to NLP, so I figured I should start small.</p>
  <p>Today, by chance, I stumbled on this paper <a href="https://arxiv.org/abs/1607.01759" rel="nofollow noreferrer">Bag of Tricks for Efficient Text Classification</a>, which very well may be the inspiration for <code>nn.EmbeddingBag</code>. Regardless, I read the paper and saw that they increased performance through using <em>&quot;n-grams as additional features to capture some partial information about the local word order&quot;</em></p>
  <p>So by the wording of this sentence, specifically &quot;additional features&quot;, I take it to mean that they made n-grams as part of their vocabulary. For example &quot;abc news&quot; is treated as a single word in the vocabulary, and then appended to the training data that is being embedded like so:</p>
  <pre><code>dataset = TextFromPandas(tweet_df)
  label, sentence, ngrams = dataset[0]
  label, sentence, ngrams
  
  # out:
  
  (1,
   'quake our deeds are the reason of this # earthquake may allah forgive us all',
   ['quake our',
    'our deeds',
    'deeds are',
    'are the',
    'the reason',
    'reason of',
    'of this',
    'this #',
    '# earthquake',
    'earthquake may',
    'may allah',
    'allah forgive',
    'forgive us',
    'us all'])
  </code></pre>
  <p>I just wanted to check my assumption, because the paper is not very explicit. I already tried to string n-grams together as a new sentence in place of the old, but performance dropped significantly.</p>
  <p>I will continue to experiment, but I was wondering if anyone knows the specific mechanism?</p>
  
Tags: <natural-language-processing><papers><feature-extraction><bag-of-words><n-gram>
Title: Bag of Tricks: n-grams as additional features?
UserDisplayName: rocksNwaves
ThreadText: # Bag of Tricks: n-grams as additional features?
  
  (Asked by: username_0 on 2021-08-26)
  
  I've been playing with PyTorch's `nn.EmbeddingBag` for sentence classification for about a month. I've been doing some feature engineering, playing with different tokenizers, etc. I'm just trying to get the best performance out of this simple model as I can. I'm new to NLP, so I figured I should start small.
  
  Today, by chance, I stumbled on this paper [Bag of Tricks for Efficient Text Classification](https://arxiv.org/abs/1607.01759), which very well may be the inspiration for `nn.EmbeddingBag`. Regardless, I read the paper and saw that they increased performance through using *"n-grams as additional features to capture some partial information about the local word order"*
  
  So by the wording of this sentence, specifically "additional features", I take it to mean that they made n-grams as part of their vocabulary. For example "abc news" is treated as a single word in the vocabulary, and then appended to the training data that is being embedded like so:
  
  ```
  dataset = TextFromPandas(tweet_df)
  label, sentence, ngrams = dataset[0]
  label, sentence, ngrams
  
  # out:
  
  (1,
   'quake our deeds are the reason of this # earthquake may allah forgive us all',
   ['quake our',
    'our deeds',
    'deeds are',
    'are the',
    'the reason',
    'reason of',
    'of this',
    'this #',
    '# earthquake',
    'earthquake may',
    'may allah',
    'allah forgive',
    'forgive us',
    'us all'])
  
  ```
  
  I just wanted to check my assumption, because the paper is not very explicit. I already tried to string n-grams together as a new sentence in place of the old, but performance dropped significantly.
  
  I will continue to experiment, but I was wondering if anyone knows the specific mechanism?
  
  **Tags:** natural-language-processing, papers, feature-extraction, bag-of-words, n-gram
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-08-28. Score: 2 | Accepted answer
  
  Yes, N-grams is about *joining* $n$ words as one single token.
  
  Keep in mind it will greatly increase your features size:
  
  If you originally have 1000 unique words, notice you could get up to 1000² 2-gram (usually you don't get ALL the combinations, but notice the number of features can potentially grow huge!)
  
  If your dataset contains between thousands or millions samples, it could be enough to train a simple bag-of-words. But when you use a bi-gram, you'll probably need at least a million samples and lot more training steps. Besides, you'll probably have to tweak the hyperparameters.
  
  That's a common Machine Learning trade-off. A simple model is not very accurate. But a more complex model requires more data, more training and can overfit more easily.
  
  > **Comments:**
  > **username_0**: It seems I am constantly facing a lack of data to train the kind of models I'd like to learn.  
  > **username_1**: Welcome to machine learning!  
  
  ---
  
  