Body: <p>I am testing multiple models on <a href="https://www.kaggle.com/datasets/pavansubhasht/ibm-hr-analytics-attrition-dataset" rel="nofollow noreferrer">IBM HR Analytics Attrition Dataset</a> (1470 lines) and <a href="https://www.kaggle.com/datasets/giripujar/hr-analytics" rel="nofollow noreferrer">HR Analytics dataset</a> (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), tree-based models (Random Forest, XGBoost, CatBoost), and recently-developed deep models (TabNet, FTTransformer).</p>
  <p>Since there are many models, the first strategy I thought of to compare multiple models (to be fair for them) is to use the default hyperparameters (set by all the corresponding libraries) for all those models. But the performance on the first dataset is not good (as expected when using the default), so I need to tune hyperparameters in those models. The problem here is that different models have different types of hyperparameters, and I don't know if a fair strategy exists to fine-tune and compare the models?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-25T20:55:02.280Z
Comments.Text: I think no one as a method to avoid running multiple grid search with extensive combinations. Already comparing the default values is a good comparison. Even more valuable is to don't stop at the raw performance/metrics score, in my experience they are usually totally useless for practical applications. Instead analyzing in dept examples of different false positives and negatives coming from different models is the way to get a real understading of different models, i.e. how they fails and try to guess why, to understand what is better for what application.
Comments.UserDisplayName: Edoardo Guerriero
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-29T04:32:30.243Z
Comments.Text: @EdoardoGuerriero Thank you, that is my next strategy too, analyzing the true/false positives and negatives is very important indeed. I have tried additional experiments and concluded that since some models still perform very poorly and 1-2 models are already good, maybe an extensive grid search on some poor models is necessary.
Comments.UserDisplayName: Đào Minh Dũng
Tags: <machine-learning><hyperparameter-optimization><models>
Title: How should I compare multiple machine learning models to be (generally) fair to all models?
UserDisplayName: Đào Minh Dũng
ThreadText: # How should I compare multiple machine learning models to be (generally) fair to all models?
  
  (Asked by: username_0 on 2022-06-25)
  
  I am testing multiple models on [IBM HR Analytics Attrition Dataset](https://www.kaggle.com/datasets/pavansubhasht/ibm-hr-analytics-attrition-dataset) (1470 lines) and [HR Analytics dataset](https://www.kaggle.com/datasets/giripujar/hr-analytics) (15000 lines) for a research project. The models include traditional models (Naive Bayes, SVM), tree-based models (Random Forest, XGBoost, CatBoost), and recently-developed deep models (TabNet, FTTransformer).
  
  Since there are many models, the first strategy I thought of to compare multiple models (to be fair for them) is to use the default hyperparameters (set by all the corresponding libraries) for all those models. But the performance on the first dataset is not good (as expected when using the default), so I need to tune hyperparameters in those models. The problem here is that different models have different types of hyperparameters, and I don't know if a fair strategy exists to fine-tune and compare the models?
  
  **Tags:** machine-learning, hyperparameter-optimization, models
  
  > **Comments:**
  > **username_1**: I think no one as a method to avoid running multiple grid search with extensive combinations. Already comparing the default values is a good comparison. Even more valuable is to don't stop at the raw performance/metrics score, in my experience they are usually totally useless for practical applications. Instead analyzing in dept examples of different false positives and negatives coming from different models is the way to get a real understading of different models, i.e. how they fails and try to guess why, to understand what is better for what application.  
  > **username_0**: @username_1 Thank you, that is my next strategy too, analyzing the true/false positives and negatives is very important indeed. I have tried additional experiments and concluded that since some models still perform very poorly and 1-2 models are already good, maybe an extensive grid search on some poor models is necessary.  
  
  ---
  
  ## 0 answers
  
  