Answers.Body: <p>Yes, you are correct, <span class="math-container">$S_{dW}$</span>, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in <span class="math-container">$S_{dW}$</span> or you can say &quot;element-wise division&quot;.</p>
  <p>To avoid the zero variable in <span class="math-container">$S_{dW}$</span>, the equation usually like this:</p>
  <p><span class="math-container">$$
  W=W-\alpha \frac{dW}{\sqrt{S_{dW}+\epsilon}} \text{  where } \epsilon=10^{-8}
  $$</span></p>
  <p>If you want to implement it manually with Python, you can take a look at <a href="https://gist.github.com/acl21/64213c8f9f65bc362d595c7bec359548#file-rmsprop-py" rel="nofollow noreferrer">this link</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-18T01:05:54.000Z
Answers.LastActivityDate: 2020-12-18T01:05:54.000Z
Answers.UserDisplayName: CuCaRot
Body: <p>I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me.</p>
  <p>Here is a screenshot from <a href="https://www.youtube.com/watch?v=_e-LFe_igno&amp;ab_channel=DeepLearningAI" rel="nofollow noreferrer">this video</a> by Andrew Ng.</p>
  <p><a href="https://i.sstatic.net/HdmQP.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/HdmQP.png" alt="enter image description here" /></a></p>
  <p>From the element-wise comment, from what I understand, <span class="math-container">$dW$</span> and <span class="math-container">$db$</span> are matrices, so that must mean that <span class="math-container">$S_{dW}$</span> is a matrix (or tensor) as well.</p>
  <p>So, in the update rule, do they divide a matrix by another matrix? From what I saw on google, no such action exists.</p>
  
Tags: <deep-learning><optimization><gradient-descent><optimizers>
Title: In the update rule of RMSprop, do we divide by a matrix?
UserDisplayName: Uriyasama
ThreadText: # In the update rule of RMSprop, do we divide by a matrix?
  
  (Asked by: username_0 on 2020-12-17)
  
  I've been trying to understand RMSprop for a long time, but there's something that keeps eluding me.
  
  Here is a screenshot from [this video](https://www.youtube.com/watch?v=_e-LFe_igno&ab_channel=DeepLearningAI) by Andrew Ng.
  
  [![enter image description here](https://i.sstatic.net/HdmQP.png)](https://i.sstatic.net/HdmQP.png)
  
  From the element-wise comment, from what I understand, $dW$ and $db$ are matrices, so that must mean that $S\_{dW}$ is a matrix (or tensor) as well.
  
  So, in the update rule, do they divide a matrix by another matrix? From what I saw on google, no such action exists.
  
  **Tags:** deep-learning, optimization, gradient-descent, optimizers
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-12-18. Score: 1
  
  Yes, you are correct, $S\_{dW}$, in this case, is the matrix/tensor and it will have the same shape as gradient matrix/tensor. The update rule divides each element in gradient to each gradient in $S\_{dW}$ or you can say "element-wise division".
  
  To avoid the zero variable in $S\_{dW}$, the equation usually like this:
  
  $$
  W=W-\alpha \frac{dW}{\sqrt{S\_{dW}+\epsilon}} \text{ where } \epsilon=10^{-8}
  $$
  
  If you want to implement it manually with Python, you can take a look at [this link](https://gist.github.com/acl21/64213c8f9f65bc362d595c7bec359548#file-rmsprop-py)
  
  
  ---
  
  