Answers.Body: <p>Depending on the number of features you have you might want to try reducing features in order to reduce overfitting and speed up your model. I assume you are using proper regularization as well. PCA may be an option to help if time is the main issue, <a href="https://medium.com/@sadatnazrul/the-dos-and-donts-of-principal-component-analysis-7c2e9dc8cc48" rel="nofollow noreferrer">medium PCA</a>. As the medium article states PCA can be used to reduce the dimensionality of the data while keeping retaining variance. If you run a random forest on the different sets and the feature importance ranking is at all different that is a big issue and the number of features should be reduced. Ideally, the features should be cropped till they have the same (structure/feature importance) but slightly different weights associated with those features. Here is a <a href="https://towardsdatascience.com/feature-selection-using-random-forest-26d7b747597f" rel="nofollow noreferrer">article</a> demonstrating feature selection.</p>
  <p>Here are some publications I believe are relevant to your problem. More in line with your desire to try to validate after one sweep. You may want to look up applications of LSTM or Yolo to your model I have a feeling these technologies will help direct your final decisions on what to do.</p>
  <p><a href="https://ieeexplore.ieee.org/abstract/document/771195" rel="nofollow noreferrer">Subspace regularization method for the single-trial estimation of evoked potentials</a> (1999) by P.A. Karjalainen, J.P. Kaipio, A.S. Koistinen, M. Vauhkonen</p>
  <p><a href="https://arxiv.org/abs/1712.03847" rel="nofollow noreferrer">On Quadratic Penalties in Elastic Weight Consolidation</a> (2017) by Ferenc Huszár</p>
  <p><a href="https://dl.acm.org/citation.cfm?id=1015367" rel="nofollow noreferrer">Parameter space exploration with Gaussian process trees</a> (2014) by  Robert B. Gramacy   University of California, Santa Cruz, CA
  Herbert K. H. Lee   University of California, Santa Cruz, CA
  William G. Macready</p>
  <p><a href="https://arxiv.org/abs/1707.06799" rel="nofollow noreferrer">Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks</a> (2017) by Nils Reimers, Iryna Gurevych</p>
  <p><a href="https://arxiv.org/abs/1804.01825" rel="nofollow noreferrer">Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio</a> (2018) by Alexei Botchkarev -evaluation of regression performance</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-13T21:43:17.900Z
Answers.LastActivityDate: 2021-03-31T22:20:14.917Z
Answers.LastEditDate: 2021-03-31T22:20:14.917Z
Answers.UserDisplayName: Michael Hearn
Body: <p>Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our results. Instead, we do 10 fold cross validation (CV) to estimate how well the model might generalize.</p>
  <p>We're training deep learning models that take between 24-48 hours each, and the process of parameter sweeping is obviously very slow when performing 10-fold cross validation.</p>
  <p>Does anyone have any experience or citations for how well parameter sweeping on one split of the data followed by cross validation (used to estimate how well it generalizes) works?</p>
  <p>I suspect it's highly dependent on the distribution of data and local minima &amp; maxima of the hyper parameters, but I wanted to ask.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-09-25T13:41:29.997Z
Comments.Text: One problem: you might slightly overfit when testing on the set that the parameter sweep was tested on. A more rigorous approach is to test the parameter sweep on a validation set, which is held out during cross validation. But you would still end up testing on the set that you trained the parameter sweep on, which has the potential to overfit.
Comments.UserDisplayName: Reinstate Monica
Tags: <machine-learning><deep-learning><hyperparameter-optimization><cross-validation><generalization>
Title: Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?
UserDisplayName: Philipp Cannons
ThreadText: # Will parameter sweeping on one split of data followed by cross validation discover the right hyperparameters?
  
  (Asked by: username_0 on 2019-09-25)
  
  Let's call our dataset splits train/test/evaluate. We're in a situation where we require months of data. So we prefer to use the evaluation dataset as infrequently as possible to avoid polluting our results. Instead, we do 10 fold cross validation (CV) to estimate how well the model might generalize.
  
  We're training deep learning models that take between 24-48 hours each, and the process of parameter sweeping is obviously very slow when performing 10-fold cross validation.
  
  Does anyone have any experience or citations for how well parameter sweeping on one split of the data followed by cross validation (used to estimate how well it generalizes) works?
  
  I suspect it's highly dependent on the distribution of data and local minima & maxima of the hyper parameters, but I wanted to ask.
  
  **Tags:** machine-learning, deep-learning, hyperparameter-optimization, cross-validation, generalization
  > **Comments:**
  > **username_1**: One problem: you might slightly overfit when testing on the set that the parameter sweep was tested on. A more rigorous approach is to test the parameter sweep on a validation set, which is held out during cross validation. But you would still end up testing on the set that you trained the parameter sweep on, which has the potential to overfit.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-11-13. Score: 0
  
  Depending on the number of features you have you might want to try reducing features in order to reduce overfitting and speed up your model. I assume you are using proper regularization as well. PCA may be an option to help if time is the main issue, [medium PCA](https://medium.com/@sadatnazrul/the-dos-and-donts-of-principal-component-analysis-7c2e9dc8cc48). As the medium article states PCA can be used to reduce the dimensionality of the data while keeping retaining variance. If you run a random forest on the different sets and the feature importance ranking is at all different that is a big issue and the number of features should be reduced. Ideally, the features should be cropped till they have the same (structure/feature importance) but slightly different weights associated with those features. Here is a [article](https://towardsdatascience.com/feature-selection-using-random-forest-26d7b747597f) demonstrating feature selection.
  
  Here are some publications I believe are relevant to your problem. More in line with your desire to try to validate after one sweep. You may want to look up applications of LSTM or Yolo to your model I have a feeling these technologies will help direct your final decisions on what to do.
  
  [Subspace regularization method for the single-trial estimation of evoked potentials](https://ieeexplore.ieee.org/abstract/document/771195) (1999) by P.A. Karjalainen, J.P. Kaipio, A.S. Koistinen, M. Vauhkonen
  
  [On Quadratic Penalties in Elastic Weight Consolidation](https://arxiv.org/abs/1712.03847) (2017) by Ferenc Huszár
  
  [Parameter space exploration with Gaussian process trees](https://dl.acm.org/citation.cfm?id=1015367) (2014) by Robert B. Gramacy University of California, Santa Cruz, CA
  Herbert K. H. Lee University of California, Santa Cruz, CA
  William G. Macready
  
  [Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks](https://arxiv.org/abs/1707.06799) (2017) by Nils Reimers, Iryna Gurevych
  
  [Evaluating Hospital Case Cost Prediction Models Using Azure Machine Learning Studio](https://arxiv.org/abs/1804.01825) (2018) by Alexei Botchkarev -evaluation of regression performance
  
  
  ---
  
  