NAMSG: An Efficient Method for Training Neural Networks
We introduce NAMSG, an adaptive first-order algorithm for training neural networks. The method is efficient in computation and memory, and is straightforward to implement. It computes the gradients at configurable remote observation points, in order to expedite the convergence by adjusting the step size for directions with different curvatures in the stochastic setting. It also scales the updating vector elementwise by a nonincreasing preconditioner to take the advantages of AMSGRAD. We analyze the convergence properties for both convex and nonconvex problems by modeling the training process as a dynamic system, and provide a strategy to select the observation factor without grid search. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting. The method can further achieve a O(log(T)) regret bound for strongly convex functions. Experiments demonstrate that NAMSG works well in practical problems and compares favorably to popular adaptive methods, such as ADAM, NADAM, and AMSGRAD.
This paper proposes new optimization method called NAMSG by combining AMSGRAD and NAS and introducing more parameters. There are two regret bounds in convex and strongly-convex settings and some experimental validation on MNIST and CIFAR compared to various previous methods. The purpose of the paper is unclear, and should be rejected. If the NAMSG is the next best optimization method for neural nets, then any regret bounds on convex functions is irrelevant, and the authors should focus on rigorous experimental comparisons. Currently it is not clear at all how parameters for each optimization method is set, and that the previous optimization methods were given a fair chance. There is not enough information about the experimental design to convince people that the authors were careful to rule out alternative explanations. For example, it definitely should say how the various hyperparameters are picked in various methods. On the other hand, if any of the analysis is novel, it would be good if the author can emphasize it and say what is the innovation in analysis and its significance. Right now it seems that the new parameters are plugged into a standard recipe to derive a very complicated formula, which is not helpful. I am not familiar enough with the area to tell if there is any interesting novelties in the analysis.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
NAMSG: An Efficient Method for Training Neural Networks
We introduce NAMSG, an adaptive first-order algorithm for training neural networks. The method is efficient in computation and memory, and is straightforward to implement. It computes the gradients at configurable remote observation points, in order to expedite the convergence by adjusting the step size for directions with different curvatures in the stochastic setting. It also scales the updating vector elementwise by a nonincreasing preconditioner to take the advantages of AMSGRAD. We analyze the convergence properties for both convex and nonconvex problems by modeling the training process as a dynamic system, and provide a strategy to select the observation factor without grid search. A data-dependent regret bound is proposed to guarantee the convergence in the convex setting. The method can further achieve a O(log(T)) regret bound for strongly convex functions. Experiments demonstrate that NAMSG works well in practical problems and compares favorably to popular adaptive methods, such as ADAM, NADAM, and AMSGRAD.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



This paper proposes new optimization method called NAMSG by combining AMSGRAD and NAS and introducing more parameters. There are two regret bounds in convex and strongly-convex settings and some experimental validation on MNIST and CIFAR compared to various previous methods. The purpose of the paper is unclear, and should be rejected. If the NAMSG is the next best optimization method for neural nets, then any regret bounds on convex functions is irrelevant, and the authors should focus on rigorous experimental comparisons. Currently it is not clear at all how parameters for each optimization method is set, and that the previous optimization methods were given a fair chance. There is not enough information about the experimental design to convince people that the authors were careful to rule out alternative explanations. For example, it definitely should say how the various hyperparameters are picked in various methods. On the other hand, if any of the analysis is novel, it would be good if the author can emphasize it and say what is the innovation in analysis and its significance. Right now it seems that the new parameters are plugged into a standard recipe to derive a very complicated formula, which is not helpful. I am not familiar enough with the area to tell if there is any interesting novelties in the analysis.
assistant
This paper proposes new optimization method called NAMSG by combining AMSGRAD and NAS and introducing more parameters. There are two regret bounds in convex and strongly-convex settings and some experimental validation on MNIST and CIFAR compared to various previous methods. The purpose of the paper is unclear, and should be rejected. If the NAMSG is the next best optimization method for neural nets, then any regret bounds on convex functions is irrelevant, and the authors should focus on rigorous experimental comparisons. Currently it is not clear at all how parameters for each optimization method is set, and that the previous optimization methods were given a fair chance. There is not enough information about the experimental design to convince people that the authors were careful to rule out alternative explanations. For example, it definitely should say how the various hyperparameters are picked in various methods. On the other hand, if any of the analysis is novel, it would be good if the author can emphasize it and say what is the innovation in analysis and its significance. Right now it seems that the new parameters are plugged into a standard recipe to derive a very complicated formula, which is not helpful. I am not familiar enough with the area to tell if there is any interesting novelties in the analysis.