Machine Learning, Deep Learning, Neural Networks, Optimization Algorithms, Adam.
consequence, it would be risky to use a large learning rate as it could lead to disconvergence. Momentum Based on the example above, it would be desirable to make a loss function performing larger steps in the horizontal direction and smaller steps in the vertical. This way, the convergence would