Answers.Body: <p>First of all to be more real, you usually expect more than 50% validation accuracy on articles predictions.</p>
  
  <p>Back on your question, you should definitely try to automate this process if you are looking for a long-term solution of labeling articles. Deploying such model should not cost more than hiring employees to do this manually, at least for a long-term perspective.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T14:41:43.887Z
Answers.LastActivityDate: 2019-10-23T14:41:43.887Z
Answers.UserDisplayName: theonekeyg
Answers.Body: <p>There are several advantages:</p>
  
  <ol>
  <li>Some text classification systems are <em>much</em> more accurate than 50%. For example, most spam classification systems are 99.9% accurate, or more. There will be little value to having employees review these labels.</li>
  <li>Many text classification systems can output a <em>confidence</em> as well as a label. You can selectively have employees review only the examples the model is not confident about. Often these will be small in number.</li>
  <li>You can usually <em>test</em> a text classification model by having it classify some unseen data, and then asking people to check the work. If you do this for a small number of examples, you can make sure the system is working. You can then confidently use the system on a much large set of unlabeled examples, and be reasonably sure about how accurate it is. </li>
  <li>For text, it is also important to measure how much different <em>people</em> agree on the ratings. You are unlikely to do better than this, because this gives you a notion of the subjectivity of the specific problem you are working on. If people disagree 50% of the time anyway, maybe you can accept a 50% failure rate from the automated system, and not bother checking its work.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T16:08:09.727Z
Answers.Text: I don't think that the spam classification system associated with my mail client is so accurate. For example, yesterday, it's labeled two e-mails as junk, while it was not the case.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T17:26:44.090Z
Answers.Text: @nbro You are correct, I had misremembered the statistics. It's been at 99.9% or higher for quite a long time (more than a decade). Even though the classifiers keep improving, spam authors keep using more sophisticated techniques, so it has the nature of an arms race.
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T19:32:45.877Z
Answers.Text: @nbro: You also need to account for the fact that the majority of the email you receive is likely to be spam. The spam that you correctly don't see counts for most of that 99.9% accuracy rating.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T19:57:17.973Z
Answers.Text: @NeilSlater If the majority of the e-mail I receive is spam, then I do not expect two e-mails to be marked (on the same day) as spam when they are not, if the predictive system is really understanding something about the difference between spam and not spam. In other words, if 99% of the e-mails are spam, then you can just predict always spam and you'll get a nice accuracy. Clearly, this accuracy is highly misleading.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T23:48:44.923Z
Answers.Text: @nbro FWIW, most spam classification literature uses AUC as a measurement. They typically have 99.99% AUC. This means that they can obtain extremely high accuracy on _both_ classes, despite the class imbalance
Answers.UserDisplayName: John Doucette
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-24T12:20:53.930Z
Answers.Text: I believe that point 2 actually answers my question the most. I have also come across a method that uses the predictions of multiple models. If those models all produce the same prediction, then it is likely that one can treat that prediction as 'correct' and thus save cost.
Answers.UserDisplayName: Air Christmas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-23T14:46:56.503Z
Answers.LastActivityDate: 2019-10-23T17:25:19.900Z
Answers.LastEditDate: 2019-10-23T17:25:19.900Z
Answers.UserDisplayName: John Doucette
Body: <p>(I apologize for the title being too broad and the question being not 'technical')</p>
  
  <p>Suppose that my task is to label news articles. This means that given a news article, I am supposed to classify which category that news belong to. Eg, 'Ronaldo scores a fantastic goal' should classify under 'Sports'.</p>
  
  <p>After much experimentation, I came up with a model that does this labeling for me. It has, say, 50% validation accuracy. (Assume that it is the best)</p>
  
  <p>And so I deployed this model for my task (on unseen data obviously). Of course, from a probabilistic perspective, I should get roughly 50% of the articles labelled correctly. But how do I know that which labels are actually correct and which labels need to be corrected? If I were to manually check (say, by hiring people to do so), how is deploying such a model better than just hiring people to do the classification directly? (Do not forget that the manpower cost of developing the model could have been saved.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-24T00:51:53.697Z
Comments.Text: Welcome to SE:AI!
Comments.UserDisplayName: DukeZhou
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-24T02:04:30.410Z
Comments.Text: "I should get roughly 50% of the articles labelled correctly." You should? If there are n categories, each article fitting one category, and articles are evenly distributed across categories, random luck would give you a 1/n chance of getting it correct for one article. If it's just sports vs non-sports, (again evenly distributed), sure, 50% is no better than random luck, but that doesn't seem to be the case.
Comments.UserDisplayName: muru
Tags: <classification><utility>
Title: How does text classification reduce manpower costs?
UserDisplayName: Air Christmas
ThreadText: # How does text classification reduce manpower costs?
  
  (Asked by: username_0 on 2019-10-23)
  
  (I apologize for the title being too broad and the question being not 'technical')
  
  Suppose that my task is to label news articles. This means that given a news article, I am supposed to classify which category that news belong to. Eg, 'Ronaldo scores a fantastic goal' should classify under 'Sports'.
  
  After much experimentation, I came up with a model that does this labeling for me. It has, say, 50% validation accuracy. (Assume that it is the best)
  
  And so I deployed this model for my task (on unseen data obviously). Of course, from a probabilistic perspective, I should get roughly 50% of the articles labelled correctly. But how do I know that which labels are actually correct and which labels need to be corrected? If I were to manually check (say, by hiring people to do so), how is deploying such a model better than just hiring people to do the classification directly? (Do not forget that the manpower cost of developing the model could have been saved.)
  
  **Tags:** classification, utility
  > **Comments:**
  > **username_1**: Welcome to SE:AI!  
  > **username_2**: "I should get roughly 50% of the articles labelled correctly." You should? If there are n categories, each article fitting one category, and articles are evenly distributed across categories, random luck would give you a 1/n chance of getting it correct for one article. If it's just sports vs non-sports, (again evenly distributed), sure, 50% is no better than random luck, but that doesn't seem to be the case.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_4 on 2019-10-23. Score: 10 | Accepted answer
  
  There are several advantages:
  
  1. Some text classification systems are *much* more accurate than 50%. For example, most spam classification systems are 99.9% accurate, or more. There will be little value to having employees review these labels.
  2. Many text classification systems can output a *confidence* as well as a label. You can selectively have employees review only the examples the model is not confident about. Often these will be small in number.
  3. You can usually *test* a text classification model by having it classify some unseen data, and then asking people to check the work. If you do this for a small number of examples, you can make sure the system is working. You can then confidently use the system on a much large set of unlabeled examples, and be reasonably sure about how accurate it is.
  4. For text, it is also important to measure how much different *people* agree on the ratings. You are unlikely to do better than this, because this gives you a notion of the subjectivity of the specific problem you are working on. If people disagree 50% of the time anyway, maybe you can accept a 50% failure rate from the automated system, and not bother checking its work.
  
  > **Comments:**
  > **username_5**: I don't think that the spam classification system associated with my mail client is so accurate. For example, yesterday, it's labeled two e-mails as junk, while it was not the case.  
  > **username_4**: @nbro You are correct, I had misremembered the statistics. It's been at 99.9% or higher for quite a long time (more than a decade). Even though the classifiers keep improving, spam authors keep using more sophisticated techniques, so it has the nature of an arms race.  
  > **username_6**: @nbro: You also need to account for the fact that the majority of the email you receive is likely to be spam. The spam that you correctly don't see counts for most of that 99.9% accuracy rating.  
  > **username_5**: @NeilSlater If the majority of the e-mail I receive is spam, then I do not expect two e-mails to be marked (on the same day) as spam when they are not, if the predictive system is really understanding something about the difference between spam and not spam. In other words, if 99% of the e-mails are spam, then you can just predict always spam and you'll get a nice accuracy. Clearly, this accuracy is highly misleading.  
  > **username_4**: @nbro FWIW, most spam classification literature uses AUC as a measurement. They typically have 99.99% AUC. This means that they can obtain extremely high accuracy on _both_ classes, despite the class imbalance  
  > **username_0**: I believe that point 2 actually answers my question the most. I have also come across a method that uses the predictions of multiple models. If those models all produce the same prediction, then it is likely that one can treat that prediction as 'correct' and thus save cost.  
  
  ---
  
  ### Answer by username_3 on 2019-10-23. Score: 0
  
  First of all to be more real, you usually expect more than 50% validation accuracy on articles predictions.
  
  Back on your question, you should definitely try to automate this process if you are looking for a long-term solution of labeling articles. Deploying such model should not cost more than hiring employees to do this manually, at least for a long-term perspective.
  
  
  ---
  
  