Answers.Body: <p>Since learning rate is a hyperparameter, using different learning rates or epochs for each model is not only allowed but generally encouraged as long as the hyperparameter tuning process such as grid search is consistent across all models. Different models have unique architectures, parameter counts, and training dynamics, which can affect how they respond to specific learning rates. Comparing models using the same fixed learning rate may disadvantage certain models, as they might not reach their optimal performance under those conditions.</p>
  <p>Finally it's important to keep other factors constant such as the training dataset, evaluation metrics, data splits, preprocessing steps, etc. Of course, if resources are permitted, you can also run all models with the same learning rate and training setup to explore how they perform under identical conditions which could potentially provide additional insights.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-21T03:45:48.807Z
Answers.Text: Thank you for your answer, I appreciate your help. I found some research paper who use fixed learning rate to compare between models for that i'm wondering why they do that or is it  a must to compare in a fair situation?, also do you think is it better to use learning rate schedualer in this situation?
Answers.UserDisplayName: Balira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-21T04:14:21.620Z
Answers.Text: @Balira They must have some other specific reason which you may not notice or they may not disclose. Most optimization of ML models cannot control (global) learning rate, for example, Adam as the most populate optimizer of ANNs could be said to be a parameter-specific implicit learning rate scheduler which is not controlled by any external global learning rate scheduler.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-21T04:22:15.410Z
Answers.Text: Thank you cinch, you really provide me with important points
Answers.UserDisplayName: Balira
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-16T18:51:00.843Z
Answers.LastActivityDate: 2024-11-16T18:51:00.843Z
Answers.UserDisplayName: cinch
Body: <p>If I'm using the same datasets  and using different language models, such as BERT, DistilBERT, Electra, etc., and my goal to compare the performance, such as the accuracy, macro f score, etc. should I used the same learning rate and number of epochs to make comparison fair, or is it allowed to have different learning rates for each? For example, if BERT provides the best result with a learning rate 2e-5 and Electra provides the best results with 1e-6, can I register both results and use them?</p>
  
Tags: <deep-learning><transformer><language-model><learning-rate>
Title: Should I use the same learning rate when I compare the performance for different language models?
UserDisplayName: Balira
ThreadText: # Should I use the same learning rate when I compare the performance for different language models?
  
  (Asked by: username_0 on 2024-11-16)
  
  If I'm using the same datasets and using different language models, such as BERT, DistilBERT, Electra, etc., and my goal to compare the performance, such as the accuracy, macro f score, etc. should I used the same learning rate and number of epochs to make comparison fair, or is it allowed to have different learning rates for each? For example, if BERT provides the best result with a learning rate 2e-5 and Electra provides the best results with 1e-6, can I register both results and use them?
  
  **Tags:** deep-learning, transformer, language-model, learning-rate
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-16. Score: 1 | Accepted answer
  
  Since learning rate is a hyperparameter, using different learning rates or epochs for each model is not only allowed but generally encouraged as long as the hyperparameter tuning process such as grid search is consistent across all models. Different models have unique architectures, parameter counts, and training dynamics, which can affect how they respond to specific learning rates. Comparing models using the same fixed learning rate may disadvantage certain models, as they might not reach their optimal performance under those conditions.
  
  Finally it's important to keep other factors constant such as the training dataset, evaluation metrics, data splits, preprocessing steps, etc. Of course, if resources are permitted, you can also run all models with the same learning rate and training setup to explore how they perform under identical conditions which could potentially provide additional insights.
  
  > **Comments:**
  > **username_0**: Thank you for your answer, I appreciate your help. I found some research paper who use fixed learning rate to compare between models for that i'm wondering why they do that or is it  a must to compare in a fair situation?, also do you think is it better to use learning rate schedualer in this situation?  
  > **username_1**: @Balira They must have some other specific reason which you may not notice or they may not disclose. Most optimization of ML models cannot control (global) learning rate, for example, Adam as the most populate optimizer of ANNs could be said to be a parameter-specific implicit learning rate scheduler which is not controlled by any external global learning rate scheduler.  
  > **username_0**: Thank you cinch, you really provide me with important points  
  
  ---
  
  