Answers.Body: <p>You should not use the <strong>training data</strong> for hyper-parameter tuning. In other words, when doing the hyper-parameter tuning, you should not optimize the training objective. You should optimize an objective computed on a dataset that is different than the training dataset. This dataset is sometimes called <strong>validation dataset</strong>, which can also be used for <strong>early stopping</strong>, which is not a way of hyper-parameter tuning but to avoid <strong>over-fitting</strong>.</p>
  <p>In the <a href="https://keras.io/keras_tuner/" rel="nofollow noreferrer">Keras Tuner</a>, you can specify the validation data (which is passed to the <code>fit</code> method under the hood) and the objective of the hyper-parameter optimization. You should specify that the objective is computed on the validation data (e.g. <code>val_loss</code> or <code>val_accuracy</code>), which should be different than the training data. <a href="https://keras.io/guides/keras_tuner/getting_started/" rel="nofollow noreferrer">Here</a> you have a complete example.</p>
  <p>Once you have selected your hyper-parameters, you can train again the <strong>best model</strong> on a training dataset, which can or not be the same as the training dataset you used during hyper-parameter optimization, but you should use a <strong>test dataset</strong> to assess the <strong>generalization</strong> ability of your model. The main requirement is that the training, validation, and test datasets are disjoint in order to avoid bias.</p>
  <p>If you use <strong>k-fold cross-validation</strong>, you will be training and testing your model with different parts of your whole dataset each time. So, if you have <span class="math-container">$k$</span> folds, you will use <span class="math-container">$k - 1$</span> folds for training and one for testing. You will do this <span class="math-container">$k$</span> times, each time with a different fold for testing and the rest for training. This means that you will be using all your data for training and testing, but, each time, the training and test datasets are separate. You use k-fold CV in order to compute an average (and/or variance) estimate of generalisation of your model (you do this especially when your dataset is small).</p>
  <p>In principle, I don't see any big problem with doing k-fold cross-validation after you have selected the best hyper-parameters (e.g. architecture), as long as you used a separate validation dataset, which is disjoint from the data you used for k-fold CV. However, make sure to shuffle your data before hyper-parameter tuning (i.e. before selecting the validation dataset), so that the distribution of the training, validation, and test datasets is roughly the same.</p>
  <p>If I understand your descriptions correctly, it seems that you used the validation data also during k-fold CV. I don't think this was a good idea, as you're training and testing a model using data that you used to select that model itself (so this can introduce some kind of bias). You also don't specify which metric you optimize during hyper-parameter optimization.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T10:54:58.743Z
Answers.Text: thank you! this answers my question. so would you recommend just throwing away the validation data used for hyperparameter tuning during k-fold cross val? 
  
  so for example, if my training set for HP tuning is 10 total samples with 1 sample used for validation, would you simply remove that one sample for k-fold and train on 9 samples?
Answers.UserDisplayName: user9317212
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T10:57:19.520Z
Answers.Text: @user9317212 Yes, I would recommend that you remove the validation data before doing the k-fold CV. However, if your dataset is really that small, I don't know if it even makes sense to perform HP optimization with 1 sample.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T11:02:39.117Z
Answers.Text: great! thank you! no - it is not, but i wanted to use the most simple example just be concrete about my point -
Answers.UserDisplayName: user9317212
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-08T10:45:48.950Z
Answers.LastActivityDate: 2021-12-08T10:45:48.950Z
Answers.UserDisplayName: nbro
Body: <p>I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network.</p>
  <p>Is it valid to do this (determine the best hyper-parameters) and <strong>then</strong> do cross-validation to get a more accurate test estimation of the dataset?</p>
  <p>I don't see how this would be invalid, given that the cross-validation examples I have seen already have network architectures known a priori, presumably because this is what they chose or feel is the best way of proceeding.</p>
  <p>For hyperparameter tuning, all data is split into training and test sets - the training set is further split, when fitting the model, for a 10% validation set - the optimal model is then used to predict on the test set.</p>
  <p>For k-fold cross-validation, all data (same as above) is used, but I just split (with sklearn) the data into training and test datasets (so no validation dataset). The test set is used to determine the model performance at each iteration of k-fold cross-validation.</p>
  
Tags: <neural-networks><hyperparameter-optimization><cross-validation><multi-label-classification><k-fold-cv>
Title: Is it valid to implement hyper-parameter tuning and THEN cross-validation?
UserDisplayName: user9317212
ThreadText: # Is it valid to implement hyper-parameter tuning and THEN cross-validation?
  
  (Asked by: username_0 on 2021-06-28)
  
  I have a multi-label classification task I am solving. I have done hyperparameter tuning (with Keras Tuner) to determine the best configuration for my neural network.
  
  Is it valid to do this (determine the best hyper-parameters) and **then** do cross-validation to get a more accurate test estimation of the dataset?
  
  I don't see how this would be invalid, given that the cross-validation examples I have seen already have network architectures known a priori, presumably because this is what they chose or feel is the best way of proceeding.
  
  For hyperparameter tuning, all data is split into training and test sets - the training set is further split, when fitting the model, for a 10% validation set - the optimal model is then used to predict on the test set.
  
  For k-fold cross-validation, all data (same as above) is used, but I just split (with sklearn) the data into training and test datasets (so no validation dataset). The test set is used to determine the model performance at each iteration of k-fold cross-validation.
  
  **Tags:** neural-networks, hyperparameter-optimization, cross-validation, multi-label-classification, k-fold-cv
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-12-08. Score: 1 | Accepted answer
  
  You should not use the **training data** for hyper-parameter tuning. In other words, when doing the hyper-parameter tuning, you should not optimize the training objective. You should optimize an objective computed on a dataset that is different than the training dataset. This dataset is sometimes called **validation dataset**, which can also be used for **early stopping**, which is not a way of hyper-parameter tuning but to avoid **over-fitting**.
  
  In the [Keras Tuner](https://keras.io/keras_tuner/), you can specify the validation data (which is passed to the `fit` method under the hood) and the objective of the hyper-parameter optimization. You should specify that the objective is computed on the validation data (e.g. `val_loss` or `val_accuracy`), which should be different than the training data. [Here](https://keras.io/guides/keras_tuner/getting_started/) you have a complete example.
  
  Once you have selected your hyper-parameters, you can train again the **best model** on a training dataset, which can or not be the same as the training dataset you used during hyper-parameter optimization, but you should use a **test dataset** to assess the **generalization** ability of your model. The main requirement is that the training, validation, and test datasets are disjoint in order to avoid bias.
  
  If you use **k-fold cross-validation**, you will be training and testing your model with different parts of your whole dataset each time. So, if you have $k$ folds, you will use $k - 1$ folds for training and one for testing. You will do this $k$ times, each time with a different fold for testing and the rest for training. This means that you will be using all your data for training and testing, but, each time, the training and test datasets are separate. You use k-fold CV in order to compute an average (and/or variance) estimate of generalisation of your model (you do this especially when your dataset is small).
  
  In principle, I don't see any big problem with doing k-fold cross-validation after you have selected the best hyper-parameters (e.g. architecture), as long as you used a separate validation dataset, which is disjoint from the data you used for k-fold CV. However, make sure to shuffle your data before hyper-parameter tuning (i.e. before selecting the validation dataset), so that the distribution of the training, validation, and test datasets is roughly the same.
  
  If I understand your descriptions correctly, it seems that you used the validation data also during k-fold CV. I don't think this was a good idea, as you're training and testing a model using data that you used to select that model itself (so this can introduce some kind of bias). You also don't specify which metric you optimize during hyper-parameter optimization.
  
  > **Comments:**
  > **username_0**: thank you! this answers my question. so would you recommend just throwing away the validation data used for hyperparameter tuning during k-fold cross val?   so for example, if my training set for HP tuning is 10 total samples with 1 sample used for validation, would you simply remove that one sample for k-fold and train on 9 samples?  
  > **username_1**: @user9317212 Yes, I would recommend that you remove the validation data before doing the k-fold CV. However, if your dataset is really that small, I don't know if it even makes sense to perform HP optimization with 1 sample.  
  > **username_0**: great! thank you! no - it is not, but i wanted to use the most simple example just be concrete about my point -  
  
  ---
  
  