Answers.Body: <p>If you used your five <span class="math-container">$X_{test}$</span> sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. <em>optimizer, learning rate, batch size, dropout, activation</em>) then <strong>yes, you successfully conducted hyper-parameter optimization</strong>. However, the AUC you received for the best set of hyperparameters found (by manual tuning) <strong>is not representative of the real performance of your model</strong>.</p>
  <p>This is because the act of using a <em>test</em> set to tune the parameters of your model degenerates it back to a <em>&quot;training&quot;</em> set, because the data is not being used to measure the performance of the model but to improve it instead (although on a different level of abstraction, i.e. not to directly influence the parameters of the model, such as neural network weights), making the resulting AUC an overly optimistic biased estimator of the real AUC (that it could have resulted in for an unseen test dataset).</p>
  <p>That's why, if you care both about hyperparameter optimisation <strong>and</strong> being able to measure the &quot;real&quot; performance of your model, you need to split your dataset into three &quot;buckets&quot;: training set <span class="math-container">$X_{train}$</span>, validation st <span class="math-container">$X_{val}$</span> and test set <span class="math-container">$X_{test}$</span>. <span class="math-container">$X_{test}$</span> should only be used once (after you've trained and tuned the model), and assuming it has enough samples and the samples are representative of the unseen data, you should get a good estimate of your model performance. <span class="math-container">$X_{val}$</span>, on the other hand, is your <em>validation</em> set which you can reuse as many times as you want to find the optimal set of hyperparameters that result in the highest performance (i.e. AUC).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-16T05:49:42.807Z
Answers.Text: thanks for your answer, please provide some research references if possible
Answers.UserDisplayName: bit_scientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-06-10T21:50:11.137Z
Answers.LastActivityDate: 2023-10-11T09:18:18.427Z
Answers.LastEditDate: 2023-10-11T09:18:18.427Z
Answers.UserDisplayName: Tomasz Bartkowiak
Body: <p>I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that</p>
  
  <ul>
  <li>randomly divided the data-set into 5 folds;</li>
  <li>copy-pasted all 5-fold data-set to have 5 exact copies of data-set (folder_1 to folder_5, all absolutely same data-set)</li>
  <li>first fold in folder_1 is saved as <code>test</code> folder and remaining (fold_2, fold_3, fold_4, fold_5) are combined as one <code>train</code> folder</li>
  <li>second fold in folder_2 is saved as <code>test</code> folder and remaining (namely, fold_1, fold_3, fold_4, fold_5) are combined as one <code>train</code> folder</li>
  <li>third fold in folder_3 is saved as <code>test</code> folder and remaining (namely, fold_1, fold_2, fold_4, fold_5) are combined as one <code>train</code> folder.</li>
  <li>similar process has been done on folder_4 and foder_5.</li>
  </ul>
  
  <p>I hope, by now, you got the idea of how I distributed the data-set.</p>
  
  <p>The reason I did so is as follows:</p>
  
  <p>I have augmented the training data (<code>train</code> folder) in each of the folders and used <code>test</code> folders respectively to evaluate (ROC-AUC score). Now I kind of have 5 ROC-AUC scores which I evaluated using <code>test</code> folders. If I get the average value out of those 5 scores.</p>
  
  <p>(Assuming the above cross-validation process is done right) If I were to perform some manual hyperparameter optimizations (like an optimizer, learning rate, batch size, dropout, activation) and perform the above <strong><em>cross-validation with data augmentation</em></strong> and find the best so-called "mean ROC-AUC", <strong>does it mean I successfully conducted hyper-parameter optimization</strong>?</p>
  
  <p><strong>FYI:</strong> I have no problem with computing power OR/AND time at all to loop through the hyper-parameters for this type of <strong><em>cross-validation with data augmentation</em></strong></p>
  
Tags: <deep-learning><datasets><hyperparameter-optimization><cross-validation>
Title: How to fairly conduct a model performance with 5-fold cross validation after augmentation?
UserDisplayName: bit_scientist
ThreadText: # How to fairly conduct a model performance with 5-fold cross validation after augmentation?
  
  (Asked by: username_0 on 2020-02-17)
  
  I have, say, a (balanced) data-set with 2k images for binary classification. What I have done is that
  
  * randomly divided the data-set into 5 folds;
  * copy-pasted all 5-fold data-set to have 5 exact copies of data-set (folder\_1 to folder\_5, all absolutely same data-set)
  * first fold in folder\_1 is saved as `test` folder and remaining (fold\_2, fold\_3, fold\_4, fold\_5) are combined as one `train` folder
  * second fold in folder\_2 is saved as `test` folder and remaining (namely, fold\_1, fold\_3, fold\_4, fold\_5) are combined as one `train` folder
  * third fold in folder\_3 is saved as `test` folder and remaining (namely, fold\_1, fold\_2, fold\_4, fold\_5) are combined as one `train` folder.
  * similar process has been done on folder\_4 and foder\_5.
  
  I hope, by now, you got the idea of how I distributed the data-set.
  
  The reason I did so is as follows:
  
  I have augmented the training data (`train` folder) in each of the folders and used `test` folders respectively to evaluate (ROC-AUC score). Now I kind of have 5 ROC-AUC scores which I evaluated using `test` folders. If I get the average value out of those 5 scores.
  
  (Assuming the above cross-validation process is done right) If I were to perform some manual hyperparameter optimizations (like an optimizer, learning rate, batch size, dropout, activation) and perform the above ***cross-validation with data augmentation*** and find the best so-called "mean ROC-AUC", **does it mean I successfully conducted hyper-parameter optimization**?
  
  **FYI:** I have no problem with computing power OR/AND time at all to loop through the hyper-parameters for this type of ***cross-validation with data augmentation***
  
  **Tags:** deep-learning, datasets, hyperparameter-optimization, cross-validation
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-06-10. Score: 3 | Accepted answer
  
  If you used your five $X\_{test}$ sets multiple times (to measure the average AUC) to decide on the best set of hyperparameters (i.e. *optimizer, learning rate, batch size, dropout, activation*) then **yes, you successfully conducted hyper-parameter optimization**. However, the AUC you received for the best set of hyperparameters found (by manual tuning) **is not representative of the real performance of your model**.
  
  This is because the act of using a *test* set to tune the parameters of your model degenerates it back to a *"training"* set, because the data is not being used to measure the performance of the model but to improve it instead (although on a different level of abstraction, i.e. not to directly influence the parameters of the model, such as neural network weights), making the resulting AUC an overly optimistic biased estimator of the real AUC (that it could have resulted in for an unseen test dataset).
  
  That's why, if you care both about hyperparameter optimisation **and** being able to measure the "real" performance of your model, you need to split your dataset into three "buckets": training set $X\_{train}$, validation st $X\_{val}$ and test set $X\_{test}$. $X\_{test}$ should only be used once (after you've trained and tuned the model), and assuming it has enough samples and the samples are representative of the unseen data, you should get a good estimate of your model performance. $X\_{val}$, on the other hand, is your *validation* set which you can reuse as many times as you want to find the optimal set of hyperparameters that result in the highest performance (i.e. AUC).
  
  > **Comments:**
  > **username_0**: thanks for your answer, please provide some research references if possible  
  
  ---
  
  