Machine Learning, Loss Function, Optimisation, Lua, Deep Learning.
applied to all parameter updates. If we have sparse data, we may want to update the parameters in different extent instead. Adaptive gradient descent algorithms such as Adagrad, Adadelta, RMSprop, Adam, provide an alternative to classical SGD. They have per-paramter learning rate methods, which