We provide a theoretical explanation for the effectiveness of gradient clipping in training deep neural networks. The key ingredient is a new smoothness condition derived from practical neural network training examples. We observe that gradient smoothness, a concept central to the analysis of first-order optimization algorithms that is often assumed to be a constant, demonstrates significant variability along the training trajectory of deep neural networks. Further, this smoothness positively correlates with the gradient norm, and contrary to standard assumptions in the literature, it can grow with the norm of the gradient. These empirical observations limit the applicability of existing theoretical analyses of algorithms that rely on a fixed bound on smoothness. These observations motivate us to introduce a novel relaxation of gradient smoothness that is weaker than the commonly used Lipschitz smoothness assumption. Under the new condition, we prove that two popular methods, namely, gradient clipping and normalized gradient, converge arbitrarily faster than gradient descent with fixed stepsize. We further explain why such adaptively scaled gradient methods can accelerate empirical convergence and verify our results empirically in popular neural network training settings.
In this paper, the authors relax the generally used Lipschitz smoothness condition in optimization, to a more general smoothness condition that may depend on norm of the gradient. The authors proved that, with this relaxed condition, under such cases, both GD and clipped GD can converge within O(1/\epsilon^2) time, but when their exists x where the gradient norm can be very large, the clipped GD will converge provably faster than GD with a proved GD convergence rate lower bound. The authors also generalize their results to SGD. Experiments show that in deep neural network local gradient Lipschitz constant is scale with gradient norm, and use clipped gradient can accelerate convergence while keep good generalization performance as expected, which is well observed by other researchers.
Detailed comments:
1. The f^* In theorem 3 is not defined. I think it’s not the f^* in Assumption 1? Should be more clear. Also, in non-convex optimization, can the deterministic GD find the global optimum? If f^* is the stationary point the algorithm find, then the Assumption 1 is a little weird. The authors should better clarify this.
2. Feel the claim of the Assumption can be relaxed to only hold on \mathcal{S} is a little confusing. Which Assumption can be relaxed? It seems that in the proof of Lemma 9, we need the Assumption 3 holds for the set \|x^+-x\| \leq min{1/L_1, 1}. I think maybe the other Assumption will also be needed globally, unless the authors can prove that the optimization is only on a compact set in S.
3. Maybe a discussion connected the existing Lipschitz constant based results to the new results in this paper can make the readers more aware of the contribution in this paper. For example, the Lipschitz constant of neural network can be of order O(L_1 M) due to the definition, so if L_1 M is large, for clipped GD we can still have expected decreasing of O(1/L_0) each turn while vanilla GD with the best O(1/L) step size can still only have O(1/L) where L = L_0 + L_1 M due to the traditional results based on Lipschitz smoothness.
I don’t totally go through the whole proof, but I think the results are reasonable and the most of the techniques are standard to the whole community. So there may be no fatal error that violates the conclusion. I find the whole paper interesting and match the intuition from the practice. However, I think it can be polished to make the whole idea more clear and more easy to accept by the potential theorists and practitioner audience.