AdamT: A Stochastic Optimization with Trend Correction Scheme
Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing for capability on large-scale sparse datasets. On top of that, they are computationally efficient and insensitive to the hyper-parameter settings. In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT. Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients. The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly. We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings.
The experimental results of this paper are not convincing. i) One should use much more powerful networks. ii) Different variants of Adam should be considered, also with different learning rate schedules. iii) One should study how newly introduced hyperparameters affect the results of AdamT. I find it hard to trust your conclusions like *Empirical results demonstrate that our method, AdamT, works well in practice and constantly beats the baseline method Adam* after inspecting Figure 3. Even much greater difference is performance can be due to a small change of hyperparameters, here you have several new hyperparameters.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
AdamT: A Stochastic Optimization with Trend Correction Scheme
Adam-typed optimizers, as a class of adaptive moment estimation methods with the exponential moving average scheme, have been successfully used in many applications of deep learning. Such methods are appealing for capability on large-scale sparse datasets. On top of that, they are computationally efficient and insensitive to the hyper-parameter settings. In this paper, we present a new framework for adapting Adam-typed methods, namely AdamT. Instead of applying a simple exponential weighted average, AdamT also includes the trend information when updating the parameters with the adaptive step size and gradients. The newly added term is expected to efficiently capture the non-horizontal moving patterns on the cost surface, and thus converge more rapidly. We show empirically the importance of the trend component, where AdamT outperforms the conventional Adam method constantly in both convex and non-convex settings.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The experimental results of this paper are not convincing. i) One should use much more powerful networks. ii) Different variants of Adam should be considered, also with different learning rate schedules. iii) One should study how newly introduced hyperparameters affect the results of AdamT. I find it hard to trust your conclusions like *Empirical results demonstrate that our method, AdamT, works well in practice and constantly beats the baseline method Adam* after inspecting Figure 3. Even much greater difference is performance can be due to a small change of hyperparameters, here you have several new hyperparameters.
assistant
The experimental results of this paper are not convincing. i) One should use much more powerful networks. ii) Different variants of Adam should be considered, also with different learning rate schedules. iii) One should study how newly introduced hyperparameters affect the results of AdamT. I find it hard to trust your conclusions like *Empirical results demonstrate that our method, AdamT, works well in practice and constantly beats the baseline method Adam* after inspecting Figure 3. Even much greater difference is performance can be due to a small change of hyperparameters, here you have several new hyperparameters.