Answers.Body: <p>A learning rate of 1 is indeed atypical in many machine learning contexts, as it may cause the model to converge too quickly and overshoot the global minimum.</p>
  <p>Generally, a smaller learning rate within the range (0,1) is preferred to allow the model to learn gradually and avoid missing the global minimum. However, the optimal learning rate can depend on the specific architecture and dataset.</p>
  <p>If a high learning rate like 1 is working for your GNN with GAT layers, it may indicate that your model and dataset have certain unique characteristics. However, it's also possible that your model could benefit from a lower learning rate and more training epochs.</p>
  <p>You may want to experiment with different learning rates and observe the effects on your model's training and validation performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-15T17:33:45.403Z
Answers.LastActivityDate: 2023-11-15T17:33:45.403Z
Answers.UserDisplayName: Ari Setiawan
Answers.Body: <p>From an optimization POV, there is no problem, indeed, given a loss <span class="math-container">$L$</span> for which a good learning is <span class="math-container">$10^{-4}$</span>, you can build a second loss equivalent which has <span class="math-container">$1$</span> as good learning rate, just by multiplying by a scalar <span class="math-container">$L' 10^{-4} L$</span> (<span class="math-container">$\nabla L' = 10^{-4}\nabla L$</span>)</p>
  <p>In DL might happen that, for example, during regression, the targets and the predictions are very small, so that &quot;scalar&quot; that i was talking about, is hidden in the targets</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T21:39:36.923Z
Answers.LastActivityDate: 2023-11-16T21:39:36.923Z
Answers.UserDisplayName: Alberto
Answers.Body: <p>Learning rate must be below 1 because if it were above 1 the output would only depend on the last input of the last training epoch , which cant be correct.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-17T15:09:44.047Z
Answers.Text: Hi @Root and welcome to AI Stack Exchange! If possible, maybe add some details to help explain this answer to OP. Thank you for joining, and we look forward to seeing you around this site more in the near future!
Answers.UserDisplayName: DeepQZero
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T21:45:19.577Z
Answers.LastActivityDate: 2023-11-16T21:45:19.577Z
Answers.UserDisplayName: Root
Body: <p>As the title says my GNN with three layers of GAT (Graph attention layers) is only moving the metrics when the learning rate is 1. As generally the learning rate is (0,1) should I be worried?</p>
  <p>Also here it says that if lr is larger than 1 it is solely focusing on the gradient instead of the model parameters. Don't know if this is a good or bad thing.
  <a href="https://ai.stackexchange.com/questions/23740/why-is-the-learning-rate-generally-beneath-1">Why is the learning rate generally beneath 1?</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-16T14:13:50.753Z
Comments.Text: Maybe you have extremely small gradients that force you to use a large lr. Could you check the gradient's norm? Also, it would be helpful for us to see a couple of learning curves each denoting a different learning rate (e.g., 1, 0.01, etc.)
Comments.UserDisplayName: Luca Anzalone
Tags: <deep-learning><training><learning-rate><gnn>
Title: My model is only improving when learning rate is 1. Should I be worried?
UserDisplayName: DataDoge
ThreadText: # My model is only improving when learning rate is 1. Should I be worried?
  
  (Asked by: username_0 on 2023-11-15)
  
  As the title says my GNN with three layers of GAT (Graph attention layers) is only moving the metrics when the learning rate is 1. As generally the learning rate is (0,1) should I be worried?
  
  Also here it says that if lr is larger than 1 it is solely focusing on the gradient instead of the model parameters. Don't know if this is a good or bad thing.
  [Why is the learning rate generally beneath 1?](https://ai.stackexchange.com/questions/23740/why-is-the-learning-rate-generally-beneath-1)
  
  **Tags:** deep-learning, training, learning-rate, gnn
  
  > **Comments:**
  > **username_1**: Maybe you have extremely small gradients that force you to use a large lr. Could you check the gradient's norm? Also, it would be helpful for us to see a couple of learning curves each denoting a different learning rate (e.g., 1, 0.01, etc.)  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_2 on 2023-11-15. Score: 1
  
  A learning rate of 1 is indeed atypical in many machine learning contexts, as it may cause the model to converge too quickly and overshoot the global minimum.
  
  Generally, a smaller learning rate within the range (0,1) is preferred to allow the model to learn gradually and avoid missing the global minimum. However, the optimal learning rate can depend on the specific architecture and dataset.
  
  If a high learning rate like 1 is working for your GNN with GAT layers, it may indicate that your model and dataset have certain unique characteristics. However, it's also possible that your model could benefit from a lower learning rate and more training epochs.
  
  You may want to experiment with different learning rates and observe the effects on your model's training and validation performance.
  
  
  ---
  
  ### Answer by username_3 on 2023-11-16. Score: 0
  
  From an optimization POV, there is no problem, indeed, given a loss $L$ for which a good learning is $10^{-4}$, you can build a second loss equivalent which has $1$ as good learning rate, just by multiplying by a scalar $L' 10^{-4} L$ ($\nabla L' = 10^{-4}\nabla L$)
  
  In DL might happen that, for example, during regression, the targets and the predictions are very small, so that "scalar" that i was talking about, is hidden in the targets
  
  
  ---
  
  ### Answer by username_4 on 2023-11-16. Score: 0
  
  Learning rate must be below 1 because if it were above 1 the output would only depend on the last input of the last training epoch , which cant be correct.
  
  > **Comments:**
  > **username_5**: Hi @username_4 and welcome to AI Stack Exchange! If possible, maybe add some details to help explain this answer to OP. Thank you for joining, and we look forward to seeing you around this site more in the near future!  
  
  ---
  
  