Answers.Body: <p>This is a standard ML problem: changing hyper-parameters changes the performance of the whole model. Ideally, you'd be cross-validating hyper-parameter choices, not merely comparing on a static validation set. That being said you need to be careful with hyper-parameter optimization because you could overfit these to the peculiarities of your validation set ; cross-validation helps to some extent but really what helps is having a test set that you hardly ever test against. Ideally never before you've chosen your HPs with (cross-) validation. And test-set performance will indicate how much your  HP-optimization procedure was biased.</p>
  
  <p>I'm afraid training from scratch is your only solution. This does not however mean that you have to train until the end and many hyper-parameter optimization techniques out there will help you stop training early enough so you don't waste computational resources on HPs which are not worth it. A good starting point would be this blog post by Criteo's Aloïs Bissuel: <a href="https://medium.com/criteo-labs/hyper-parameter-optimization-algorithms-2fe447525903" rel="nofollow noreferrer">Hyper-parameter optimization algorithms: a short review</a> </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-14T22:18:16.697Z
Answers.LastActivityDate: 2020-04-14T22:18:16.697Z
Answers.UserDisplayName: ted
Answers.Body: <p>The importance of having a totally separate test set is very crucial. Once you start to use the validation set performance as a measure to use to tune hyper parameters you are biasing your network to work well on the validation set so it can no longer be relied on as a true measure of performance. Eventually if you use your test set too often then adjust hyper parameters to improve performance on the test set you wind up in the same boat. I have actually used several test sets to try to avoid this trap.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-15T05:25:38.973Z
Answers.LastActivityDate: 2020-04-15T05:25:38.973Z
Answers.UserDisplayName: Gerry P
Body: <p>I've built a neural network from the scratch, choosing arbitrary numbers for the hyperparameters: learning rate, number of hidden layers and neurons for these, number of epochs and size of mini batches. Now that I've been able to build something potentially useful (~93% of accuracy with test data, unseen by the model before), I want to focus on hyperparameter tuning.</p>
  
  <p>The conceptual difference between training and validation sets is clear and makes a lot of sense. It's obvious that the model is biased towards the training set, so it wouldn't make sense to use it to tune the hyperparameters, nor for evaluating its performance.</p>
  
  <p>But, how can I use the validation set for this, if changing any of the parameters enforces me to rebuild a new model again? The final prediction depends on the values of X number of MxN matrices (weights) and X number of N vectors (biases), whose values depend on the learning rate, batch size and number of epochs; and whose dimensions depends on the number and size of hidden layers. If I change any of these, I'd need to rebuild my model again. So I'd be using this validation set for training different models, ending up as in the first step: fitting a model from the scratch.</p>
  
  <p>To sum up: I fall in a recursive problem in which I need to fine tune the hyperparameters of my model with unseen data, but changing any of these hyperparameters implies rebuilding the model.</p>
  
Tags: <neural-networks><machine-learning><hyperparameter-optimization>
Title: How is a validation set used to tune the hyperparameters in a non-biased way, if the new models depends on the values of these?
UserDisplayName: Julen
ThreadText: # How is a validation set used to tune the hyperparameters in a non-biased way, if the new models depends on the values of these?
  
  (Asked by: username_0 on 2020-04-14)
  
  I've built a neural network from the scratch, choosing arbitrary numbers for the hyperparameters: learning rate, number of hidden layers and neurons for these, number of epochs and size of mini batches. Now that I've been able to build something potentially useful (~93% of accuracy with test data, unseen by the model before), I want to focus on hyperparameter tuning.
  
  The conceptual difference between training and validation sets is clear and makes a lot of sense. It's obvious that the model is biased towards the training set, so it wouldn't make sense to use it to tune the hyperparameters, nor for evaluating its performance.
  
  But, how can I use the validation set for this, if changing any of the parameters enforces me to rebuild a new model again? The final prediction depends on the values of X number of MxN matrices (weights) and X number of N vectors (biases), whose values depend on the learning rate, batch size and number of epochs; and whose dimensions depends on the number and size of hidden layers. If I change any of these, I'd need to rebuild my model again. So I'd be using this validation set for training different models, ending up as in the first step: fitting a model from the scratch.
  
  To sum up: I fall in a recursive problem in which I need to fine tune the hyperparameters of my model with unseen data, but changing any of these hyperparameters implies rebuilding the model.
  
  **Tags:** neural-networks, machine-learning, hyperparameter-optimization
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2020-04-14. Score: 1 | Accepted answer
  
  This is a standard ML problem: changing hyper-parameters changes the performance of the whole model. Ideally, you'd be cross-validating hyper-parameter choices, not merely comparing on a static validation set. That being said you need to be careful with hyper-parameter optimization because you could overfit these to the peculiarities of your validation set ; cross-validation helps to some extent but really what helps is having a test set that you hardly ever test against. Ideally never before you've chosen your HPs with (cross-) validation. And test-set performance will indicate how much your HP-optimization procedure was biased.
  
  I'm afraid training from scratch is your only solution. This does not however mean that you have to train until the end and many hyper-parameter optimization techniques out there will help you stop training early enough so you don't waste computational resources on HPs which are not worth it. A good starting point would be this blog post by Criteo's Aloïs Bissuel: [Hyper-parameter optimization algorithms: a short review](https://medium.com/criteo-labs/hyper-parameter-optimization-algorithms-2fe447525903)
  
  
  ---
  
  ### Answer by username_2 on 2020-04-15. Score: 1
  
  The importance of having a totally separate test set is very crucial. Once you start to use the validation set performance as a measure to use to tune hyper parameters you are biasing your network to work well on the validation set so it can no longer be relied on as a true measure of performance. Eventually if you use your test set too often then adjust hyper parameters to improve performance on the test set you wind up in the same boat. I have actually used several test sets to try to avoid this trap.
  
  
  ---
  
  