Body: <p>I have been trying to train a model with 0.001 learning rate. I tried regression techniques, early stopping and lr manipulations within epochs. But nothing felt right even though after numerous tries i got an &quot;okay&quot; result with 0.75 loss for categorical_crossentropy with around 80% accuracy.</p>
  <p>The validation loss graphs were always in a crazy spike and jittery they would not normalize on a plateu as they should. I know that this issue is about the gradient descent and higher learning rates overshoot the solution.</p>
  <p>But what is it that makes my problem not suitable for the default learning rate of keras, how can be my weights so &quot;sensitive(?)&quot; to 0.001 learning rate ?</p>
  <p>When I lowered the learning rate to 0.00005 or 0.00001 losses that I inspected were quite lower and I have not ran enough iterations yet but it seems possible to at least converge around the local minima.</p>
  <p>Can you suggest a solution for an adaptive learning rate maybe ? If such thing exists</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-03T10:14:10.497Z
Comments.Text: highly depends on which optimizer and which library you are using
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-03T10:49:44.800Z
Comments.Text: @Alberto  I’m using TensorFlow with the Keras API, and my optimizer of choice is Adam with its default parameters, except for the learning rate adjustments I mentioned.
  
  Could you suggest how different optimizers or their parameters might better suit my model, particularly considering its sensitivity to the learning rate? I’ve noticed improved stability with lower learning rates, but I’m curious if there’s an optimizer setting or perhaps a different optimizer altogether that could help optimize the convergence and stability without drastically reducing the learning rate.
Comments.UserDisplayName: Yigithan Sever
Tags: <neural-networks><keras><gradient-descent><models><learning-rate>
Title: Learning Rate greater than ~0.00005 significantly hinders model performance and increases loss
UserDisplayName: Yigithan Sever
ThreadText: # Learning Rate greater than ~0.00005 significantly hinders model performance and increases loss
  
  (Asked by: username_0 on 2024-07-03)
  
  I have been trying to train a model with 0.001 learning rate. I tried regression techniques, early stopping and lr manipulations within epochs. But nothing felt right even though after numerous tries i got an "okay" result with 0.75 loss for categorical\_crossentropy with around 80% accuracy.
  
  The validation loss graphs were always in a crazy spike and jittery they would not normalize on a plateu as they should. I know that this issue is about the gradient descent and higher learning rates overshoot the solution.
  
  But what is it that makes my problem not suitable for the default learning rate of keras, how can be my weights so "sensitive(?)" to 0.001 learning rate ?
  
  When I lowered the learning rate to 0.00005 or 0.00001 losses that I inspected were quite lower and I have not ran enough iterations yet but it seems possible to at least converge around the local minima.
  
  Can you suggest a solution for an adaptive learning rate maybe ? If such thing exists
  
  **Tags:** neural-networks, keras, gradient-descent, models, learning-rate
  > **Comments:**
  > **username_1**: highly depends on which optimizer and which library you are using  
  > **username_0**: @Alberto  I’m using TensorFlow with the Keras API, and my optimizer of choice is Adam with its default parameters, except for the learning rate adjustments I mentioned.  Could you suggest how different optimizers or their parameters might better suit my model, particularly considering its sensitivity to the learning rate? I’ve noticed improved stability with lower learning rates, but I’m curious if there’s an optimizer setting or perhaps a different optimizer altogether that could help optimize the convergence and stability without drastically reducing the learning rate.  
  
  ---
  
  ## 0 answers
  
  