Answers.Body: <p>The main purpose of a validation procedure is to obtain an accurate estimate of model performance.  In brief, in a typical hold out validation, the total dataset is split into a training and test set.  A model is then trained on a training set; however, the model's performance (e.g., accuracy) is evaluated on the test set.  The idea here is that estimating performance on the same dataset that you used to train the model is obviously going to give you a biased estimate of performance.  The performance estimate on the test set is supposed to be a more accurate / unbiased representation of how the model will perform when deployed.  However, the randomness of the train-test split may yield spurious results, especially when datasets are small.</p>
  <p>This is where cross-validation comes in.  Cross-validation repeats this procedure multiple times using different train-test splits on the same dataset.  The performance on the different test sets are then averaged over all the iterations giving a more accurate estimate (along with confidence bounds if you calculate the standard deviation) for the model's performance.  This averaging approach is much less dependent on a single train-test split, which by chance may not have been representative of the distribution of the overall dataset.</p>
  <p>Thus, to answer your question, a cross-validation generates multiple models on (overlapping) subsets of data that are not 'aware&quot; of each other.  A k-fold cross-validation, for example, will execute k iterations, generating a new model at each iteration.  This scikit learn <a href="https://scikit-learn.org/stable/modules/cross_validation.html" rel="nofollow noreferrer">user guide</a> explains cross-validation (and its many variations) very well.</p>
  <p>The cross-validation procedure, as mentioned, is used to get an accurate estimate of model performance in the real world.  It's main purpose is not to generate a final model.  The final model can be generated by training on the total dataset, but you would not then estimate accuracy by evaluating the performance of that model on the total dataset (as explained this would be a highly biased estimate).  Instead, use the average accuracy from the cross-validation.</p>
  <p>As an aside, if you decide to save these models after each iteration, you can combine them to make an ensemble model.  However, that is not technically a part of the cross-validation procedure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T13:53:44.330Z
Answers.Text: Thank you. Things start getting more clear thanks to your answer.
  Still, probably misunderstood something previously, not regarding the cross validation but the "simple" validation.
  Please, correct me if  if I am wrong in the following:
  
  A-during an epoch the weighs are adjusted each batch as I understand. The adjustion of weights after each batch is based on training set ground truth -  nothing to do with validation set here here, right? 
  
  B-If A is right, than, the validation is being done at the end of each epoch, but not with a goal to adjust weight, but in order to estimate. Right?
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T16:58:00.653Z
Answers.Text: My pleasure.  It seems like you are referring to training of neural networks here by your use of "epoch".  The same principles apply.  **A)**  and  **B)** are correct.  Generally, the data are split into train-validation-test sets for training neural networks [[1](https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7)].
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T17:04:40.023Z
Answers.Text: After each epoch, the performance of the model is assessed on the validation set.  The purpose is to assess for over/underfitting.  These articles will help you understand those concepts [[1](https://www.geeksforgeeks.org/underfitting-and-overfitting-in-machine-learning/), [2](https://machinelearningmastery.com/overfitting-and-underfitting-with-machine-learning-algorithms/), [3](https://towardsdatascience.com/overfitting-vs-underfitting-a-complete-example-d05dd7e19765)]
Answers.UserDisplayName: Snehal Patel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T18:30:47.400Z
Answers.Text: Thank you again. Your explanations are very clear.
Answers.UserDisplayName: Igor
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-08T04:50:42.297Z
Answers.LastActivityDate: 2023-01-08T17:08:40.503Z
Answers.LastEditDate: 2023-01-08T17:08:40.503Z
Answers.UserDisplayName: Snehal Patel
Body: <p>I am trying to determine the result of a cross validation process. Is it just a set of standalone models which is produced after each cross-validation round, or is there some kind of final model which is &quot;aware&quot; of all the models created by the Kfold rounds and benefits from them?</p>
  
Tags: <cross-validation>
Title: What is a result of a cross validation process?
UserDisplayName: Igor
ThreadText: # What is a result of a cross validation process?
  
  (Asked by: username_0 on 2023-01-05)
  
  I am trying to determine the result of a cross validation process. Is it just a set of standalone models which is produced after each cross-validation round, or is there some kind of final model which is "aware" of all the models created by the Kfold rounds and benefits from them?
  
  **Tags:** cross-validation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-08. Score: 1 | Accepted answer
  
  The main purpose of a validation procedure is to obtain an accurate estimate of model performance. In brief, in a typical hold out validation, the total dataset is split into a training and test set. A model is then trained on a training set; however, the model's performance (e.g., accuracy) is evaluated on the test set. The idea here is that estimating performance on the same dataset that you used to train the model is obviously going to give you a biased estimate of performance. The performance estimate on the test set is supposed to be a more accurate / unbiased representation of how the model will perform when deployed. However, the randomness of the train-test split may yield spurious results, especially when datasets are small.
  
  This is where cross-validation comes in. Cross-validation repeats this procedure multiple times using different train-test splits on the same dataset. The performance on the different test sets are then averaged over all the iterations giving a more accurate estimate (along with confidence bounds if you calculate the standard deviation) for the model's performance. This averaging approach is much less dependent on a single train-test split, which by chance may not have been representative of the distribution of the overall dataset.
  
  Thus, to answer your question, a cross-validation generates multiple models on (overlapping) subsets of data that are not 'aware" of each other. A k-fold cross-validation, for example, will execute k iterations, generating a new model at each iteration. This scikit learn [user guide](https://scikit-learn.org/stable/modules/cross_validation.html) explains cross-validation (and its many variations) very well.
  
  The cross-validation procedure, as mentioned, is used to get an accurate estimate of model performance in the real world. It's main purpose is not to generate a final model. The final model can be generated by training on the total dataset, but you would not then estimate accuracy by evaluating the performance of that model on the total dataset (as explained this would be a highly biased estimate). Instead, use the average accuracy from the cross-validation.
  
  As an aside, if you decide to save these models after each iteration, you can combine them to make an ensemble model. However, that is not technically a part of the cross-validation procedure.
  
  > **Comments:**
  > **username_0**: Thank you. Things start getting more clear thanks to your answer. Still, probably misunderstood something previously, not regarding the cross validation but the "simple" validation. Please, correct me if  if I am wrong in the following:  A-during an epoch the weighs are adjusted each batch as I understand. The adjustion of weights after each batch is based on training set ground truth -  nothing to do with validation set here here, right?   B-If A is right, than, the validation is being done at the end of each epoch, but not with a goal to adjust weight, but in order to estimate. Right?  
  > **username_1**: My pleasure.  It seems like you are referring to training of neural networks here by your use of "epoch".  The same principles apply.  **A)**  and  **B)** are correct.  Generally, the data are split into train-validation-test sets for training neural networks [[1](https://towardsdatascience.com/train-validation-and-test-sets-72cb40cba9e7)].  
  > **username_1**: After each epoch, the performance of the model is assessed on the validation set.  The purpose is to assess for over/underfitting.  These articles will help you understand those concepts [[1](https://www.geeksforgeeks.org/underfitting-and-overfitting-in-machine-learning/), [2](https://machinelearningmastery.com/overfitting-and-underfitting-with-machine-learning-algorithms/), [3](https://towardsdatascience.com/overfitting-vs-underfitting-a-complete-example-d05dd7e19765)]  
  > **username_0**: Thank you again. Your explanations are very clear.  
  
  ---
  
  