Answers.Body: <blockquote>
    <p>My question is, why they keep the documents separated and create that statistic, if it is posible to merge all documents of the same class? This would have two advantages:</p>
    
    <ul>
    <li>You can just use word counts instead of frequencies, as the documents per class label is 1.</li>
    </ul>
  </blockquote>
  
  <p>In general, I don't think this is the case. I don't know if you have a specific equation in mind where it would end up being the same thing mathematically? Anyway, in general, it is possible that some documents in your corpus are very short, and others are very long. In such cases, you'd still want to make sure to use frequencies rather than raw word counts.</p>
  
  <p>For example, suppose you have one very short text that is specifically about England. The word "England" may appear 10 times, but have a very high frequency due to it being a short text. If you compare it to a massive text that is about all countries in the world, that massive text may have the word "England" appearing 20 times, but with a significantly lower (relative) frequency.</p>
  
  <blockquote>
    <ul>
    <li>Instead of using IDF, you just select features with enough standard deviation between classes.</li>
    </ul>
  </blockquote>
  
  <p>I don't think this would work correctly because you may have significant differences among documents within a single class. Suppose, for example, that you have the following two classes of documents:</p>
  
  <ol>
  <li>Scientific articles (about AI, math, biology, linguistics, astronomy, whatever else you can think of...)</li>
  <li>News articles</li>
  </ol>
  
  <p>Each of the "subdomains" in the single "scientific articles" class would likely have some highly specific terminology they use, which could be detected through TF-IDF. However, even though they're all in the same class of "scientific articles", they're likely all quite different from each other. If you put them all together and treat them as a single document, there is a risk that they'll all "average out" and become much more difficult to distinguish from a more general class such as the class of "news articles".</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-07-25T09:23:41.213Z
Answers.LastActivityDate: 2018-07-25T11:38:21.850Z
Answers.LastEditDate: 2018-07-25T11:38:21.850Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>Most of the literature considers text classification as the classification of documents. When using the bag-of-words and Bayesian classification, they usually use the statistic TF-IDF, where TF normalizes the word count with the number of words per document, and IDF focuses on ignoring widely used and thus useless words for this task.</p>
  <p>My question is, why they keep the documents separated and create that statistic, if it is possible to merge all documents of the same class? This would have two advantages:</p>
  <ul>
  <li><p>You can just use word counts instead of frequencies, as the documents per class label is 1.</p>
  </li>
  <li><p>Instead of using IDF, you just select features with enough standard deviation between classes.</p>
  </li>
  </ul>
  
Tags: <natural-language-processing><text-classification><tf-idf><bag-of-words>
Title: Why are documents kept separated when training a text classifier?
UserDisplayName: freesoul
ThreadText: # Why are documents kept separated when training a text classifier?
  
  (Asked by: username_0 on 2018-07-24)
  
  Most of the literature considers text classification as the classification of documents. When using the bag-of-words and Bayesian classification, they usually use the statistic TF-IDF, where TF normalizes the word count with the number of words per document, and IDF focuses on ignoring widely used and thus useless words for this task.
  
  My question is, why they keep the documents separated and create that statistic, if it is possible to merge all documents of the same class? This would have two advantages:
  
  * You can just use word counts instead of frequencies, as the documents per class label is 1.
  * Instead of using IDF, you just select features with enough standard deviation between classes.
  
  **Tags:** natural-language-processing, text-classification, tf-idf, bag-of-words
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-07-25. Score: 1
  
  > My question is, why they keep the documents separated and create that statistic, if it is posible to merge all documents of the same class? This would have two advantages:
  >
  > * You can just use word counts instead of frequencies, as the documents per class label is 1.
  
  In general, I don't think this is the case. I don't know if you have a specific equation in mind where it would end up being the same thing mathematically? Anyway, in general, it is possible that some documents in your corpus are very short, and others are very long. In such cases, you'd still want to make sure to use frequencies rather than raw word counts.
  
  For example, suppose you have one very short text that is specifically about England. The word "England" may appear 10 times, but have a very high frequency due to it being a short text. If you compare it to a massive text that is about all countries in the world, that massive text may have the word "England" appearing 20 times, but with a significantly lower (relative) frequency.
  
  > * Instead of using IDF, you just select features with enough standard deviation between classes.
  
  I don't think this would work correctly because you may have significant differences among documents within a single class. Suppose, for example, that you have the following two classes of documents:
  
  1. Scientific articles (about AI, math, biology, linguistics, astronomy, whatever else you can think of...)
  2. News articles
  
  Each of the "subdomains" in the single "scientific articles" class would likely have some highly specific terminology they use, which could be detected through TF-IDF. However, even though they're all in the same class of "scientific articles", they're likely all quite different from each other. If you put them all together and treat them as a single document, there is a risk that they'll all "average out" and become much more difficult to distinguish from a more general class such as the class of "news articles".
  
  
  ---
  
  