Machine Learning, Gradient Descent, Optimization, Deep Learning, Artificial Intelligence.
Performance of Gradient Descent Variations SGD, RMS Prop, Adam and AdamW converged to the global minima within the first 20000 iterations, while AdaDelta and Adagrad converges slowly. Adagrad has the worst convergence rate. Adam : 0 at [0,0] SGD : 0 at [0,0] RMSProp : 0 at [0,0] AdaDelta : 0.94 at