<p>Yes, you are correct, <span class="math-container">$S_{dW}$</span>, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in <span class="math-container">$S_{dW}$</span> or you can say &quot;element-wise division&quot;.</p>
<p>To avoid the zero variable in <span class="math-container">$S_{dW}$</span>, the equation usually like this:</p>
<p><span class="math-container">$$
W=W-\alpha \frac{dW}{\sqrt{S_{dW}+\epsilon}} \text{  where } \epsilon=10^{-8}
$$</span></p>
<p>If you want to implement it manually with Python, you can take a look at <a href="https://gist.github.com/acl21/64213c8f9f65bc362d595c7bec359548#file-rmsprop-py" rel="nofollow noreferrer">this link</a></p>

CC BY-SA 4.0
2020-12-18T01:05:54.000Z
2020-12-18T01:05:54.000Z
CuCaRot
<p>I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me.</p>
<p>Here is a screenshot from <a href="https://www.youtube.com/watch?v=_e-LFe_igno&amp;ab_channel=DeepLearningAI" rel="nofollow noreferrer">this video</a> by Andrew Ng.</p>
<p><a href="https://i.sstatic.net/HdmQP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HdmQP.png" alt="enter image description here" /></a></p>
<p>From the element-wise comment, from what I understand, <span class="math-container">$dW$</span> and <span class="math-container">$db$</span> are matrices, so that must mean that <span class="math-container">$S_{dW}$</span> is a matrix (or tensor) as well.</p>
<p>So, in the update rule, do they divide a matrix by another matrix? From what I saw on google, no such action exists.</p>

<deep-learning><optimization><gradient-descent><optimizers>
In the update rule of RMSprop, do we divide by a matrix?
Uriyasama
# In the update rule of RMSprop, do we divide by a matrix?

(Asked by: username_0 on 2020-12-17)

I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me.

Here is a screenshot from [this video](https://www.youtube.com/watch?v=_e-LFe_igno&ab_channel=DeepLearningAI) by Andrew Ng.

[![enter image description here](https://i.sstatic.net/HdmQP.png)](https://i.sstatic.net/HdmQP.png)

From the element-wise comment, from what I understand, $dW$ and $db$ are matrices, so that must mean that $S\_{dW}$ is a matrix (or tensor) as well.

So, in the update rule, do they divide a matrix by another matrix? From what I saw on google, no such action exists.

**Tags:** deep-learning, optimization, gradient-descent, optimizers


---

## 1 answers

### Answer by username_1 on 2020-12-18. Score: 1

Yes, you are correct, $S\_{dW}$, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in $S\_{dW}$ or you can say "element-wise division".

To avoid the zero variable in $S\_{dW}$, the equation usually like this:

$$
W=W-\alpha \frac{dW}{\sqrt{S\_{dW}+\epsilon}} \text{ where } \epsilon=10^{-8}
$$

If you want to implement it manually with Python, you can take a look at [this link](https://gist.github.com/acl21/64213c8f9f65bc362d595c7bec359548#file-rmsprop-py)


---

