A new perspective in understanding of Adam-Type algorithms and beyond
First-order adaptive optimization algorithms such as Adam play an important role in modern deep learning due to their super fast convergence speed in solving large scale optimization problems. However, Adam*s non-convergence behavior and regrettable generalization ability make it fall into a love-hate relationship to deep learning community. Previous studies on Adam and its variants (refer as Adam-Type algorithms) mainly rely on theoretical regret bound analysis, which overlook the natural characteristic reside in such algorithms and limit our thinking. In this paper, we aim at seeking a different interpretation of Adam-Type algorithms so that we can intuitively comprehend and improve them. The way we chose is based on a traditional online convex optimization algorithm scheme known as mirror descent method. By bridging Adam and mirror descent, we receive a clear map of the functionality of each part in Adam. In addition, this new angle brings us a new insight on identifying the non-convergence issue of Adam. Moreover, we provide new variant of Adam-Type algorithm, namely AdamAL which can naturally mitigate the non-convergence issue of Adam and improve its performance. We further conduct experiments on various popular deep learning tasks and models, and the results are quite promising.
The paper proposes to study some weaknesses of Adam and AMSGrad and propose a new method called AdamAL that is evaluated on CIFAR10. I am not from this area, but unfortunately I find this paper to not be rigorously written or organized. Section 3.3 for instance which discusses the non-alignment projection issue with AMSGrad is not rigorously written. There are no proofs to any theorems and even some of the theorems/corollaries are not written rigorously. Organization-wise I feel it is difficult to see where the paper is going and some sort of outline / notation box would help. Empirically, there are results on one dataset CIFAR10 that shows the author*s proposed variant AdamAL works better. However, it is not the detailed types of experiments I was expecting of a paper that points out specific issues in AMSGrad. Shouldn*t the experiments be showing these weaknesses in some sort of controlled setting?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A new perspective in understanding of Adam-Type algorithms and beyond
First-order adaptive optimization algorithms such as Adam play an important role in modern deep learning due to their super fast convergence speed in solving large scale optimization problems. However, Adam*s non-convergence behavior and regrettable generalization ability make it fall into a love-hate relationship to deep learning community. Previous studies on Adam and its variants (refer as Adam-Type algorithms) mainly rely on theoretical regret bound analysis, which overlook the natural characteristic reside in such algorithms and limit our thinking. In this paper, we aim at seeking a different interpretation of Adam-Type algorithms so that we can intuitively comprehend and improve them. The way we chose is based on a traditional online convex optimization algorithm scheme known as mirror descent method. By bridging Adam and mirror descent, we receive a clear map of the functionality of each part in Adam. In addition, this new angle brings us a new insight on identifying the non-convergence issue of Adam. Moreover, we provide new variant of Adam-Type algorithm, namely AdamAL which can naturally mitigate the non-convergence issue of Adam and improve its performance. We further conduct experiments on various popular deep learning tasks and models, and the results are quite promising.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The paper proposes to study some weaknesses of Adam and AMSGrad and propose a new method called AdamAL that is evaluated on CIFAR10. I am not from this area, but unfortunately I find this paper to not be rigorously written or organized. Section 3.3 for instance which discusses the non-alignment projection issue with AMSGrad is not rigorously written. There are no proofs to any theorems and even some of the theorems/corollaries are not written rigorously. Organization-wise I feel it is difficult to see where the paper is going and some sort of outline / notation box would help. Empirically, there are results on one dataset CIFAR10 that shows the author*s proposed variant AdamAL works better. However, it is not the detailed types of experiments I was expecting of a paper that points out specific issues in AMSGrad. Shouldn*t the experiments be showing these weaknesses in some sort of controlled setting?
assistant
The paper proposes to study some weaknesses of Adam and AMSGrad and propose a new method called AdamAL that is evaluated on CIFAR10. I am not from this area, but unfortunately I find this paper to not be rigorously written or organized. Section 3.3 for instance which discusses the non-alignment projection issue with AMSGrad is not rigorously written. There are no proofs to any theorems and even some of the theorems/corollaries are not written rigorously. Organization-wise I feel it is difficult to see where the paper is going and some sort of outline / notation box would help. Empirically, there are results on one dataset CIFAR10 that shows the author*s proposed variant AdamAL works better. However, it is not the detailed types of experiments I was expecting of a paper that points out specific issues in AMSGrad. Shouldn*t the experiments be showing these weaknesses in some sort of controlled setting?