Artificial Intelligence, Machine Learning, Deep Learning, Neural Networks.
element-wise multiplication In the update step, the gradient is scaled down (element-wise division) by a factor of √(s+ε) (ε is a smoothening term) θ​=θ​-α∇J(θ​) divided by √(s+ε) division here is element-wise In a way, the algorithm decays the learning rate — faster for steep slopes and slower for