Answers.Body: <p>You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data. </p>
  
  <p>Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better with unseen data.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-26T10:02:43.207Z
Answers.LastActivityDate: 2019-11-26T10:02:43.207Z
Answers.UserDisplayName: Lustwelpintje
Answers.Body: <p>Assuming that your cross-validation scores(both on train set and test set) indicate model's prediction performance correctly, you should definitely decide which trained model to use based on your <strong>validation accuracy only,</strong> regardless your model is overfitted or not. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-29T18:07:45.120Z
Answers.Text: Hi, thanks for your answer. But I'm not sure if I got it entirely, can be please elaborate a bit more. From my above examples, which one would you go with and why?
Answers.UserDisplayName: Supratim Haldar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-06-29T06:20:10.943Z
Answers.LastActivityDate: 2019-06-29T06:20:10.943Z
Answers.UserDisplayName: Sang Su Lee
Body: <p>The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.</p>
  
  <p>I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:</p>
  
  <pre><code>(1) Train set: 0.99 | Cross-validation set: 0.72
  (2) Train set: 0.75 | Cross-validation set: 0.70
  (3) Train set: 0.69 | Cross-validation set: 0.69
  </code></pre>
  
  <p>These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.</p>
  
  <p>My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?</p>
  
Tags: <machine-learning><training><cross-validation>
Title: Ideal score of a model on training and cross validation data
UserDisplayName: Supratim Haldar
ThreadText: # Ideal score of a model on training and cross validation data
  
  (Asked by: username_0 on 2019-06-07)
  
  The question is little bit broad, but I could not find any concrete explanation anywhere, hence decided to ask the experts here.
  
  I have trained a classifier model for binary classification task. Now I am trying to fine tune the model. With different sets of hyperparameters I am getting different sets of accuracy on my train and test set. For example:
  
  ```
  (1) Train set: 0.99 | Cross-validation set: 0.72
  (2) Train set: 0.75 | Cross-validation set: 0.70
  (3) Train set: 0.69 | Cross-validation set: 0.69
  
  ```
  
  These are approximate numbers. But my point is - for certain set of hyperparameters I am getting more or less similar CV accuracy, while the accuracy on training data varies from overfit to not so much overfit.
  
  My question is - which of these models will work best on future unseen data? What is the recommendation in this scenario, shall we choose the model with higher training accuracy or lower training accuracy, given that CV accuracy is similar in all cases above (in fact CV score is better in the overfitted model)?
  
  **Tags:** machine-learning, training, cross-validation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-11-26. Score: 1 | Accepted answer
  
  You should only look for the cross-validation score. If this set is large enough, it will give you an accurate prediction of how your model will act for unseen data.
  
  Your case is exceptional. The fitted model which is obviously overfitted actually performs better on the cross-validation set. This means in turn that your overfitted model will perform better with unseen data.
  
  
  ---
  
  ### Answer by username_2 on 2019-06-29. Score: 0
  
  Assuming that your cross-validation scores(both on train set and test set) indicate model's prediction performance correctly, you should definitely decide which trained model to use based on your **validation accuracy only,** regardless your model is overfitted or not.
  
  > **Comments:**
  > **username_0**: Hi, thanks for your answer. But I'm not sure if I got it entirely, can be please elaborate a bit more. From my above examples, which one would you go with and why?  
  
  ---
  
  