Deep Learning, Regularisation, Optimization Algorithms, Deep Neural Networks, Gradient Descent.
as: On iteration t: Compute dW, dB on current mini-batch SdW = β2 * SdW + (1 - β2) * dW2 Vdb = β2 * Sdb + (1 - β2) * db2 Update weights W = W - α * (dW/SdW) b = b - α * (db/Sdb) The slope in the vertical direction (db in our case) is steeper, resulting in a large value of Sdb. As we want slow