Answers.Body: <p>If the application is to recall answers for a specific collection of data points.  Skip the test set, just overfit to the data.</p>
  <p>Train/test is used to measure generalizability.  Cross validation (Kfold, LOO, etc) gives a statistical measure to that generalizability which is important just in case the model got lucky on a specific test set.</p>
  <p>Finally, if one is hyperparameterizing, a validation holdout is needed.  This one ensures the tuning didn’t just game the CV.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-07T14:02:02.010Z
Answers.Text: Thanks for the answer. As I see it, CV has two uses, hyperparameter tuning (through GridSearchCV for example) as well as giving a measure for generalizability apart from the test set score. But the validation sets during Kfold for example are generated from the training set, so the validation data points are not unknown anymore, how are these contributing to the generalizability then? Also, as I understand it, a validation holdout is setting a validation set in addition to a test/train dataset, but doesn't an algorithm like Kfold already set those validation sets for each fold?
Answers.UserDisplayName: Leandro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-07T15:06:54.397Z
Answers.Text: In hyperparameter tuning there are three holdouts.  Train/test/validation. Validation is held out before CV then CV is performed on the remainder.  Since validation data is not in train or test it can provide corroborating evidence towards the generalization statistics given by CV.
Answers.UserDisplayName: foreverska
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-07T19:34:43.060Z
Answers.Text: This is all true, but I'm not sure it answers the question? OTOH, the Asker has accepted it, so :shrug:
Answers.UserDisplayName: Ben Reiniger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-09-07T13:03:36.837Z
Answers.LastActivityDate: 2024-09-07T13:03:36.837Z
Answers.UserDisplayName: foreverska
Body: <p>I am new to ML and am currently tinkering with scikit. At the moment I am using a random state both for the train_test split and a random state for my MLPClassifier, to tune hyperparameters with GridsearchCV. I am not sure if I am doing the procedure correctly, but I am trying to achieve higher accuracy by searching different parameter spaces until I get an acceptable accuracy score. Since I am doing this for one fixed random state, the results are of course very biased. That's why setting fixed random states is mostly for tuning hyperparameters, correct?</p>
  <p>My question now is, if I wanted to deploy this trained model for a specific use case, would that be bad since the results are biased because I tuned the model on one 'good' state? I assume it would be but I don't really know why. Because if the model fits the needs for my specific use case why would I not train it in a fixed 'good' state?</p>
  <p>One case I can think of where training on only one state would be bad is if I wanted to analyze a model itself, and not deploy it specifically in an application for example. Would that be correct? Because when analyzing a model itself, one important aspect is of course its ability to generalize.</p>
  
Tags: <hyperparameter-optimization><scikit-learn><randomness>
Title: Why are biased results from ML models bad?
UserDisplayName: Leandro
ThreadText: # Why are biased results from ML models bad?
  
  (Asked by: username_0 on 2024-09-07)
  
  I am new to ML and am currently tinkering with scikit. At the moment I am using a random state both for the train\_test split and a random state for my MLPClassifier, to tune hyperparameters with GridsearchCV. I am not sure if I am doing the procedure correctly, but I am trying to achieve higher accuracy by searching different parameter spaces until I get an acceptable accuracy score. Since I am doing this for one fixed random state, the results are of course very biased. That's why setting fixed random states is mostly for tuning hyperparameters, correct?
  
  My question now is, if I wanted to deploy this trained model for a specific use case, would that be bad since the results are biased because I tuned the model on one 'good' state? I assume it would be but I don't really know why. Because if the model fits the needs for my specific use case why would I not train it in a fixed 'good' state?
  
  One case I can think of where training on only one state would be bad is if I wanted to analyze a model itself, and not deploy it specifically in an application for example. Would that be correct? Because when analyzing a model itself, one important aspect is of course its ability to generalize.
  
  **Tags:** hyperparameter-optimization, scikit-learn, randomness
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-09-07. Score: 0 | Accepted answer
  
  If the application is to recall answers for a specific collection of data points. Skip the test set, just overfit to the data.
  
  Train/test is used to measure generalizability. Cross validation (Kfold, LOO, etc) gives a statistical measure to that generalizability which is important just in case the model got lucky on a specific test set.
  
  Finally, if one is hyperparameterizing, a validation holdout is needed. This one ensures the tuning didn’t just game the CV.
  
  > **Comments:**
  > **username_0**: Thanks for the answer. As I see it, CV has two uses, hyperparameter tuning (through GridSearchCV for example) as well as giving a measure for generalizability apart from the test set score. But the validation sets during Kfold for example are generated from the training set, so the validation data points are not unknown anymore, how are these contributing to the generalizability then? Also, as I understand it, a validation holdout is setting a validation set in addition to a test/train dataset, but doesn't an algorithm like Kfold already set those validation sets for each fold?  
  > **username_1**: In hyperparameter tuning there are three holdouts.  Train/test/validation. Validation is held out before CV then CV is performed on the remainder.  Since validation data is not in train or test it can provide corroborating evidence towards the generalization statistics given by CV.  
  > **username_2**: This is all true, but I'm not sure it answers the question? OTOH, the Asker has accepted it, so :shrug:  
  
  ---
  
  