Answers.Body: <p>There are a lot of ways to evaluate the performance of an ML model.  You mentioned AUROC and AUPRC.  Generally you start with the confusion matrix and derive metrics such as sensitivity, accuracy, recall, precision, etc.  You can see a good outline of them <a href="https://en.wikipedia.org/wiki/Confusion_matrix" rel="nofollow noreferrer">here</a>.  </p>
  
  <p>It seems what you are asking is a shortcut to determining how good your sentiment classification model is but there aren't any without labeled test data.  You either do this by hand or you find a test set in the world, preferably something that is well know and documented and also fits your objectives.  I recommend you read Neil Slater's answer at <a href="https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228">https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228</a>.  He gives some good advice on the subjectivity of sentiment analysis classification and points out a <a href="https://www.kaggle.com/c/si650winter11/data" rel="nofollow noreferrer">labeled data set of Tweets</a> which you might be able to use to test your classifier.</p>
  
  <p>I also found this Kaggle competition which has a test set that might be of help to you: <a href="https://www.kaggle.com/c/angry-tweets/data" rel="nofollow noreferrer">Angry Tweets</a></p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-03-02T17:01:31.093Z
Answers.Text: Thanks! I suppose I'll have to do it by hand then, for my date.
Answers.UserDisplayName: schoon
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2018-02-24T16:39:04.620Z
Answers.LastActivityDate: 2018-02-24T16:39:04.620Z
Answers.UserDisplayName: Brian O'Donnell
Body: <p>I have a model that predicts sentiment of tweets. Are there any standard procedures to evaluate such a model in terms of its output?</p>
  
  <p>I could sample the output, work out which are correctly predicted by hand, and count true and false positives and negatives but is there a better way?</p>
  
  <p>I know about test and training sets and metrics like AUROC and AUPRC which evaluate the model based on known data, but I am interested in the step afterwards when we don't know the actual values we are predicting. I could use the same metrics, I suppose, but everything would need to be done by hand.</p>
  
Tags: <machine-learning><classification>
Title: How do I statistically evaluate a ML model?
UserDisplayName: schoon
ThreadText: # How do I statistically evaluate a ML model?
  
  (Asked by: username_0 on 2018-01-18)
  
  I have a model that predicts sentiment of tweets. Are there any standard procedures to evaluate such a model in terms of its output?
  
  I could sample the output, work out which are correctly predicted by hand, and count true and false positives and negatives but is there a better way?
  
  I know about test and training sets and metrics like AUROC and AUPRC which evaluate the model based on known data, but I am interested in the step afterwards when we don't know the actual values we are predicting. I could use the same metrics, I suppose, but everything would need to be done by hand.
  
  **Tags:** machine-learning, classification
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2018-02-24. Score: 3 | Accepted answer
  
  There are a lot of ways to evaluate the performance of an ML model. You mentioned AUROC and AUPRC. Generally you start with the confusion matrix and derive metrics such as sensitivity, accuracy, recall, precision, etc. You can see a good outline of them [here](https://en.wikipedia.org/wiki/Confusion_matrix).
  
  It seems what you are asking is a shortcut to determining how good your sentiment classification model is but there aren't any without labeled test data. You either do this by hand or you find a test set in the world, preferably something that is well know and documented and also fits your objectives. I recommend you read Neil Slater's answer at <https://datascience.stackexchange.com/questions/12226/how-do-i-assess-which-sentiment-classifier-is-best-for-my-project/12228>. He gives some good advice on the subjectivity of sentiment analysis classification and points out a [labeled data set of Tweets](https://www.kaggle.com/c/si650winter11/data) which you might be able to use to test your classifier.
  
  I also found this Kaggle competition which has a test set that might be of help to you: [Angry Tweets](https://www.kaggle.com/c/angry-tweets/data)
  
  > **Comments:**
  > **username_0**: Thanks! I suppose I'll have to do it by hand then, for my date.  
  
  ---
  
  