<p>The learning rate in my model is <code>0.00001</code> and the gradients of the model is within the distribution of <code>[-0.0001, 0.0001]</code>. Is it normal?</p>

CC BY-SA 4.0
2020-02-29T12:53:17.557Z
Hi and welcome to AI SE. The gradient is usually a vector (if you have more than one parameter), so it is not a scalar. Please, clarify this.
nbro
CC BY-SA 4.0
2020-02-29T14:23:50.280Z
@nbro, Yes, that true. The value distribution is within [-0.0001, 0.0001]
GoingMyWay
CC BY-SA 4.0
2020-02-29T16:24:12.570Z
These values are not so small if you're at the end of the training. At which epoch do you have these values? Could you also provide more details about your architecture, loss function, etc.?
nbro
CC BY-SA 4.0
2020-03-02T13:44:34.177Z
Seems kinda arbitrary, really.  If anything, you might think about scaling up your loss function to make it more sensitive.  The main concern would be reducing harmful rounding due to the limited precision of floating-points.
Nat
CC BY-SA 4.0
2020-03-10T08:58:46.983Z
@nbro at the early stage of training. I am using Normal initializer with mean=0 and std=1.
GoingMyWay
<deep-learning><backpropagation><gradient-descent>
Why gradients are so small in deep learning?
GoingMyWay
# Why gradients are so small in deep learning?

(Asked by: username_0 on 2020-02-29)

The learning rate in my model is `0.00001` and the gradients of the model is within the distribution of `[-0.0001, 0.0001]`. Is it normal?

**Tags:** deep-learning, backpropagation, gradient-descent

> **Comments:**
> **username_1**: Hi and welcome to AI SE. The gradient is usually a vector (if you have more than one parameter), so it is not a scalar. Please, clarify this.  
> **username_0**: @username_1, Yes, that true. The value distribution is within [-0.0001, 0.0001]  
> **username_1**: These values are not so small if you're at the end of the training. At which epoch do you have these values? Could you also provide more details about your architecture, loss function, etc.?  
> **username_2**: Seems kinda arbitrary, really.  If anything, you might think about scaling up your loss function to make it more sensitive.  The main concern would be reducing harmful rounding due to the limited precision of floating-points.  
> **username_0**: @username_1 at the early stage of training. I am using Normal initializer with mean=0 and std=1.  

---

## 0 answers

