Answers.Body: <p>You always can run your classifier on the data it was trained for, but if you wanted to validate your classifier you need new data. Running it on training data will not reveal any bias in your classifier.</p>
  <p>The question remain, how to best separate training and validation data. Some expert may discuss this in more detail.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-05T19:36:28.883Z
Answers.Text: What about cross-validation methods like out-of-bag errors from Random Forest or other machine learning approaches? Are those generally reliable?
Answers.UserDisplayName: Marcelo Araya-Salas
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-05T14:55:36.670Z
Answers.LastActivityDate: 2022-07-05T14:55:36.670Z
Answers.UserDisplayName: WMXZ
Answers.Body: <p>If you evaluate your algorithm on the train set, you will probably overestimate its performance because you will present the classifier with sound events that it has already seen before: it is most likely to classify them correcly.
  This is a bias that you can solve by having a separate test set with new data that your classifier has never seen before.
  So you should divide your annotated subset into a train set to train your classifier and a test set to measure its performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-05T15:19:57.183Z
Answers.LastActivityDate: 2022-07-05T15:19:57.183Z
Answers.UserDisplayName: Maxime BRU
Body: <p>I'm new to machine learning methods and have terabytes of acoustic data containing odontocete clicks. I want to train a classifier to classify these clicks, so I've used a subset of this data to do so. I would now like to run this classifier on all of the data, but some colleagues are warning me that it's a bad idea to have overlapping training and testing datasets. However, I don't understand why - it seems like it's a waste to 'throw away' data that likely contains useful information. I'd be appreciative of any explanation for this. Thank you.</p>
  
Tags: <machine-learning><classifier>
Title: Why can't I run a classifier on the dataset on which it was trained?
ThreadText: # Why can't I run a classifier on the dataset on which it was trained?
  
  (Asked by: username_0 on 2022-07-05)
  
  I'm new to machine learning methods and have terabytes of acoustic data containing odontocete clicks. I want to train a classifier to classify these clicks, so I've used a subset of this data to do so. I would now like to run this classifier on all of the data, but some colleagues are warning me that it's a bad idea to have overlapping training and testing datasets. However, I don't understand why - it seems like it's a waste to 'throw away' data that likely contains useful information. I'd be appreciative of any explanation for this. Thank you.
  
  **Tags:** machine-learning, classifier
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2022-07-05. Score: 14 | Accepted answer
  
  If you evaluate your algorithm on the train set, you will probably overestimate its performance because you will present the classifier with sound events that it has already seen before: it is most likely to classify them correcly.
  This is a bias that you can solve by having a separate test set with new data that your classifier has never seen before.
  So you should divide your annotated subset into a train set to train your classifier and a test set to measure its performance.
  
  
  ---
  
  ### Answer by username_1 on 2022-07-05. Score: 6
  
  You always can run your classifier on the data it was trained for, but if you wanted to validate your classifier you need new data. Running it on training data will not reveal any bias in your classifier.
  
  The question remain, how to best separate training and validation data. Some expert may discuss this in more detail.
  
  > **Comments:**
  > **username_2**: What about cross-validation methods like out-of-bag errors from Random Forest or other machine learning approaches? Are those generally reliable?  
  
  ---
  
  