Machine Learning, Artificial Intelligence, Neural Networks, Data Science, Towards Data Science.
training run. The motivation behind Polyak averaging — gradient descent with a large learning rate is unable to converge effectively to the global minimum. Suppose in t iterations you have parameters θ¹, θ², etc. We can then take the time average of these parameters to obtain a smoother estimator