AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide a new insight into the non-convergence issue of Adam as well as other adaptive learning rate methods. We argue that there exists an inappropriate correlation between gradient and the second moment term in Adam ( is the timestep), which results in that a large gradient is likely to have small step size while a small gradient may have a large step size. We demonstrate that such unbalanced step sizes are the fundamental cause of non-convergence of Adam, and we further prove that decorrelating and will lead to unbiased step size for each gradient, thus solving the non-convergence problem of Adam. Finally, we propose AdaShift, a novel adaptive learning rate method that decorrelates and by temporal shifting, i.e., using temporally shifted gradient to calculate . The experiment results demonstrate that AdaShift is able to address the non-convergence issue of Adam, while still maintaining a competitive performance with Adam in terms of both training speed and generalization.
 Summary ------ Based on an extensive argument acoordig to which Adam potential failures are due to the positive correlation between gradient and moment estimation, the authors propose Adashift, a method in which temporal shift (and more surprisingly *spatial* shift, ie mixing of parameters) is used to ensure that moment estimation is less correlated with gradient, ensuring convergence of Adashift in pathological cases, without the efficiency cost of simpler method such as AMSGrad. An extensive analysis of a pathological counter example, introduced in Reddi et al. 2018 is analysed, before the algorithm presentation and experimental validation. Experiments shows that the algorithm has equivalent speed as Adam and sometimes false local minima, resulting in better training error, and potentially better test error. Review ------- The decorrelation idea is original and well motivated by an extensive analysis of a pathological examples. The experimental validation is thorough and convincing, and the paper is overall well written. Regarding content, the reviewer is quite dubious about the spatial decorrelation idea. ASsuming shared moment estimation for blocks of parameters is definitely meaningful from an information perspective, and has indeed been used before, but it seems to have little to do with the *decorrelation* idea. The reviewer would be curious to see a comparison with temporal-only adashift in the experiment, as the block / max operator .phi, to isolate the temporal and *spatial* effect. Regarding presentation, the reviewer*s opinion is that the paper is too long. Too much space is spent discussing an interesting yet limited counterexample, on which 5 theorems (that are simple analytical derivations) are stated. This should be summarized (and its interesting argument stated more concisely), to the benefit of the actual algorithm presentation, that should appear in the main text (algorithm 3). The spatial decorrelation method, that remains unclear to the reviewer, should be discussed more and validated more extensively. The current size of the paper is 10 pages, which is much above the ICLR average length. However, due to the novelty of the algorithm, the reviewer is in favor of accepting the paper, provided the authors can address the comments above.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
Adam is shown not being able to converge to the optimal solution in certain cases. Researchers recently propose several algorithms to avoid the issue of non-convergence of Adam, but their efficiency turns out to be unsatisfactory in practice. In this paper, we provide a new insight into the non-convergence issue of Adam as well as other adaptive learning rate methods. We argue that there exists an inappropriate correlation between gradient and the second moment term in Adam ( is the timestep), which results in that a large gradient is likely to have small step size while a small gradient may have a large step size. We demonstrate that such unbalanced step sizes are the fundamental cause of non-convergence of Adam, and we further prove that decorrelating and will lead to unbiased step size for each gradient, thus solving the non-convergence problem of Adam. Finally, we propose AdaShift, a novel adaptive learning rate method that decorrelates and by temporal shifting, i.e., using temporally shifted gradient to calculate . The experiment results demonstrate that AdaShift is able to address the non-convergence issue of Adam, while still maintaining a competitive performance with Adam in terms of both training speed and generalization.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 Summary ------ Based on an extensive argument acoordig to which Adam potential failures are due to the positive correlation between gradient and moment estimation, the authors propose Adashift, a method in which temporal shift (and more surprisingly *spatial* shift, ie mixing of parameters) is used to ensure that moment estimation is less correlated with gradient, ensuring convergence of Adashift in pathological cases, without the efficiency cost of simpler method such as AMSGrad. An extensive analysis of a pathological counter example, introduced in Reddi et al. 2018 is analysed, before the algorithm presentation and experimental validation. Experiments shows that the algorithm has equivalent speed as Adam and sometimes false local minima, resulting in better training error, and potentially better test error. Review ------- The decorrelation idea is original and well motivated by an extensive analysis of a pathological examples. The experimental validation is thorough and convincing, and the paper is overall well written. Regarding content, the reviewer is quite dubious about the spatial decorrelation idea. ASsuming shared moment estimation for blocks of parameters is definitely meaningful from an information perspective, and has indeed been used before, but it seems to have little to do with the *decorrelation* idea. The reviewer would be curious to see a comparison with temporal-only adashift in the experiment, as the block / max operator .phi, to isolate the temporal and *spatial* effect. Regarding presentation, the reviewer*s opinion is that the paper is too long. Too much space is spent discussing an interesting yet limited counterexample, on which 5 theorems (that are simple analytical derivations) are stated. This should be summarized (and its interesting argument stated more concisely), to the benefit of the actual algorithm presentation, that should appear in the main text (algorithm 3). The spatial decorrelation method, that remains unclear to the reviewer, should be discussed more and validated more extensively. The current size of the paper is 10 pages, which is much above the ICLR average length. However, due to the novelty of the algorithm, the reviewer is in favor of accepting the paper, provided the authors can address the comments above.
assistant
 Summary ------ Based on an extensive argument acoordig to which Adam potential failures are due to the positive correlation between gradient and moment estimation, the authors propose Adashift, a method in which temporal shift (and more surprisingly *spatial* shift, ie mixing of parameters) is used to ensure that moment estimation is less correlated with gradient, ensuring convergence of Adashift in pathological cases, without the efficiency cost of simpler method such as AMSGrad. An extensive analysis of a pathological counter example, introduced in Reddi et al. 2018 is analysed, before the algorithm presentation and experimental validation. Experiments shows that the algorithm has equivalent speed as Adam and sometimes false local minima, resulting in better training error, and potentially better test error. Review ------- The decorrelation idea is original and well motivated by an extensive analysis of a pathological examples. The experimental validation is thorough and convincing, and the paper is overall well written. Regarding content, the reviewer is quite dubious about the spatial decorrelation idea. ASsuming shared moment estimation for blocks of parameters is definitely meaningful from an information perspective, and has indeed been used before, but it seems to have little to do with the *decorrelation* idea. The reviewer would be curious to see a comparison with temporal-only adashift in the experiment, as the block / max operator .phi, to isolate the temporal and *spatial* effect. Regarding presentation, the reviewer*s opinion is that the paper is too long. Too much space is spent discussing an interesting yet limited counterexample, on which 5 theorems (that are simple analytical derivations) are stated. This should be summarized (and its interesting argument stated more concisely), to the benefit of the actual algorithm presentation, that should appear in the main text (algorithm 3). The spatial decorrelation method, that remains unclear to the reviewer, should be discussed more and validated more extensively. The current size of the paper is 10 pages, which is much above the ICLR average length. However, due to the novelty of the algorithm, the reviewer is in favor of accepting the paper, provided the authors can address the comments above.