Machine Learning, Deep Learning, Neural Networks, Optimization Algorithms, Adam.
motivation to adapt the learning rate to computed gradient values. There might occur situations when during training, one component of the weight vector has very large gradient values while another one has extremely small. This happens especially in cases when an infrequent model parameter appears