Answers.Body: <p>Ideally, the value of <code>k</code> should be fixed and it should not be changed throughout the model training and evaluation process. <a href="https://stats.stackexchange.com/questions/27730/choice-of-k-in-k-fold-cross-validation">Here</a> is a good discussion on how to choose the value for <code>k</code>. Once you have done that, you can use k-fold cross validation to evaluate models trained using different hyperparameters.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-30T09:42:17.643Z
Answers.Text: Hey Ganesha, thanks for the contribution. I understand the value of k should be a trade-off between variance and bias. To do so I should iterate through different values of k to find the one that best satisfies both. Would not having the hyperparameters tunned be a problem?
Answers.UserDisplayName: metc
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-30T19:47:55.683Z
Answers.Text: For every fixed k, you should perform hyperparameter tuning, since all your models with different hyperparameters would be trained on the same data and will go through the same evaluation procedure, making it easier to pick the best hyperparameters.
Answers.UserDisplayName: Ganesh Tata
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-28T03:46:22.587Z
Answers.LastActivityDate: 2022-07-28T03:46:22.587Z
Answers.UserDisplayName: Ganesh Tata
Body: <p>I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a sensitivity analysis to determine the value of k. Also, I would like to do some hyperparameter grid search for the algorithm (i.e. random forests).</p>
  <p>What would be the correct procedure? First take a random value of k and perform the hyperparameter grid search and with the correct hyperparameters do the sensitivity test for k or vice versa?</p>
  <p>Thanks in advance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-07-28T03:41:35.970Z
Comments.Text: "sensitivity analysis to determine the value of k" What do you mean by this and what is your source?
Comments.UserDisplayName: Ganesh Tata
Tags: <ai-design><hyperparameter-optimization><cross-validation>
Title: Cross Validation and hyperparameter selection correct procedure
UserDisplayName: metc
ThreadText: # Cross Validation and hyperparameter selection correct procedure
  
  (Asked by: username_0 on 2022-07-25)
  
  I am trying to run a regression supervised learning problem. The dataset is not very large and I wanted to do some cross-validation to avoid overfitting. As I have read it's important to do a sensitivity analysis to determine the value of k. Also, I would like to do some hyperparameter grid search for the algorithm (i.e. random forests).
  
  What would be the correct procedure? First take a random value of k and perform the hyperparameter grid search and with the correct hyperparameters do the sensitivity test for k or vice versa?
  
  Thanks in advance!
  
  **Tags:** ai-design, hyperparameter-optimization, cross-validation
  
  > **Comments:**
  > **username_1**: "sensitivity analysis to determine the value of k" What do you mean by this and what is your source?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-28. Score: 0
  
  Ideally, the value of `k` should be fixed and it should not be changed throughout the model training and evaluation process. [Here](https://stats.stackexchange.com/questions/27730/choice-of-k-in-k-fold-cross-validation) is a good discussion on how to choose the value for `k`. Once you have done that, you can use k-fold cross validation to evaluate models trained using different hyperparameters.
  
  > **Comments:**
  > **username_0**: Hey Ganesha, thanks for the contribution. I understand the value of k should be a trade-off between variance and bias. To do so I should iterate through different values of k to find the one that best satisfies both. Would not having the hyperparameters tunned be a problem?  
  > **username_1**: For every fixed k, you should perform hyperparameter tuning, since all your models with different hyperparameters would be trained on the same data and will go through the same evaluation procedure, making it easier to pick the best hyperparameters.  
  
  ---
  
  