Body: <p>The learning rate in my model is <code>0.00001</code> and the gradients of the model is within the distribution of <code>[-0.0001, 0.0001]</code>. Is it normal?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-29T12:53:17.557Z
Comments.Text: Hi and welcome to AI SE. The gradient is usually a vector (if you have more than one parameter), so it is not a scalar. Please, clarify this.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-29T14:23:50.280Z
Comments.Text: @nbro, Yes, that true. The value distribution is within [-0.0001, 0.0001]
Comments.UserDisplayName: GoingMyWay
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-29T16:24:12.570Z
Comments.Text: These values are not so small if you're at the end of the training. At which epoch do you have these values? Could you also provide more details about your architecture, loss function, etc.?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-02T13:44:34.177Z
Comments.Text: Seems kinda arbitrary, really.  If anything, you might think about scaling up your loss function to make it more sensitive.  The main concern would be reducing harmful rounding due to the limited precision of floating-points.
Comments.UserDisplayName: Nat
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-03-10T08:58:46.983Z
Comments.Text: @nbro at the early stage of training. I am using Normal initializer with mean=0 and std=1.
Comments.UserDisplayName: GoingMyWay
Tags: <deep-learning><backpropagation><gradient-descent>
Title: Why gradients are so small in deep learning?
UserDisplayName: GoingMyWay
ThreadText: # Why gradients are so small in deep learning?
  
  (Asked by: username_0 on 2020-02-29)
  
  The learning rate in my model is `0.00001` and the gradients of the model is within the distribution of `[-0.0001, 0.0001]`. Is it normal?
  
  **Tags:** deep-learning, backpropagation, gradient-descent
  
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE. The gradient is usually a vector (if you have more than one parameter), so it is not a scalar. Please, clarify this.  
  > **username_0**: @username_1, Yes, that true. The value distribution is within [-0.0001, 0.0001]  
  > **username_1**: These values are not so small if you're at the end of the training. At which epoch do you have these values? Could you also provide more details about your architecture, loss function, etc.?  
  > **username_2**: Seems kinda arbitrary, really.  If anything, you might think about scaling up your loss function to make it more sensitive.  The main concern would be reducing harmful rounding due to the limited precision of floating-points.  
  > **username_0**: @username_1 at the early stage of training. I am using Normal initializer with mean=0 and std=1.  
  
  ---
  
  ## 0 answers
  
  