Answers.Body: <p>&quot;Selecting the model&quot; in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data.</p>
  <p>Consider the example in the link. First you like to select the best hyperparameters of your <code>svm</code> model by <code>GridSearchCV()</code>. This is done by 4-fold CV. Now the <code>clf.best_score_</code> will be the mean cross-validated score of the best estimator (the model with the best hyperparameters). However now you used the same data for training and reporting the performance, although you used CV. Keep in mind that the folds are not independent. Therefore the hyperparameters might be too data specific, i.e. your generalization error estimates are too optimistic. Therefore we like to evaluate our final model performance outside / independent of the hyperparameter selection loop / process (the call <code>cross_val_score()</code>).</p>
  <p>In the provided plot, you can clearly see that the reported performance by <code>GridSearchCV()</code> is most of the time better than the performance reported by <code>cross_val_score()</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-06T10:17:53.160Z
Answers.LastActivityDate: 2020-07-06T10:17:53.160Z
Answers.UserDisplayName: Tinu
Body: <p>I have trouble understanding how <a href="https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html" rel="nofollow noreferrer">nested cross-validation</a> works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but <strong>why are they nested?</strong></p>
  <p>From what I understood, we need to select the model before training it, which points toward non-nested loops.</p>
  <p>Could someone please explain what's wrong (or right?) with my line of reasoning, and also explain nested cross-validation in greater detail? A representative example would be great.</p>
  
Tags: <machine-learning><comparison><hyper-parameters><cross-validation>
Title: How exactly does nested cross-validation work?
UserDisplayName: stoic-santiago
ThreadText: # How exactly does nested cross-validation work?
  
  (Asked by: username_0 on 2020-07-05)
  
  I have trouble understanding how [nested cross-validation](https://scikit-learn.org/stable/auto_examples/model_selection/plot_nested_cross_validation_iris.html) works - I understand the need for two loops (one for selecting the model, and another for training the selected model), but **why are they nested?**
  
  From what I understood, we need to select the model before training it, which points toward non-nested loops.
  
  Could someone please explain what's wrong (or right?) with my line of reasoning, and also explain nested cross-validation in greater detail? A representative example would be great.
  
  **Tags:** machine-learning, comparison, hyper-parameters, cross-validation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-07-06. Score: 1 | Accepted answer
  
  "Selecting the model" in this case refers to selecting the hyperparameters of the model. The reason to use a nested CV is simply to avoid overfitting training data.
  
  Consider the example in the link. First you like to select the best hyperparameters of your `svm` model by `GridSearchCV()`. This is done by 4-fold CV. Now the `clf.best_score_` will be the mean cross-validated score of the best estimator (the model with the best hyperparameters). However now you used the same data for training and reporting the performance, although you used CV. Keep in mind that the folds are not independent. Therefore the hyperparameters might be too data specific, i.e. your generalization error estimates are too optimistic. Therefore we like to evaluate our final model performance outside / independent of the hyperparameter selection loop / process (the call `cross_val_score()`).
  
  In the provided plot, you can clearly see that the reported performance by `GridSearchCV()` is most of the time better than the performance reported by `cross_val_score()`.
  
  
  ---
  
  