A preconditioned accelerated stochastic gradient descent algorithm
We propose a preconditioned accelerated stochastic gradient method suitable for large scale optimization. We derive sufficient convergence conditions for the minimization of convex functions using a generic class of diagonal preconditioners and provide a formal convergence proof based on a framework originally used for on-line learning. Inspired by recent popular adaptive per-feature algorithms, we propose a specific preconditioner based on the second moment of the gradient. The sufficient convergence conditions motivate a critical adaptation of the per-feature updates in order to ensure convergence. We show empirical results for the minimization of convex and non-convex cost functions, in the context of neural network training. The method compares favorably with respect to current, first order, stochastic optimization methods.
 The paper talks about a method to combine preconditioning at the per feature level and Nesterov-like acceleration for SGD optimization. The explanation of the method in Section 3 should be self-contained. The main result, computational context, etc., are poorly described, so that it would not be easily understandable to a non-expert. What was the reason for the choice of the mini batch size of 128. I would guess that you would actually see interesting differences for the method by varying this parameter. How does this compare with the FLAG method of Chen et al from AISTATS, which is motivated by similar issues and addresses similar concerns, obtaining stronger results as far as I can tell? The figures and captions and inserts are extremely hard to read, so much so that I have to trust it when the authors tell me that their results are better. The empirical evaluation for "convex problems" is for LS regression. Hmm. Is there not a better convex problem that can be used to illustrate the strength and weaknesses of the method. If not, why don*t you compare to a state-of-the-art least squares solver. For the empirical results, what looks particularly interesting is some tradeoffs, e.g, a slower initial convergence, that are shown. Given the limited scope of the empirical evaluations, it*s difficult to tell whether there is much to argue for the method. But those tradeoffs are seen in other contexts, e.g., with subsampled second order methods, and it would be good to understand those tradeoffs, since that might point to where and if a methods such as this is useful. The conclusions in the conclusion are overly broad.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
A preconditioned accelerated stochastic gradient descent algorithm
We propose a preconditioned accelerated stochastic gradient method suitable for large scale optimization. We derive sufficient convergence conditions for the minimization of convex functions using a generic class of diagonal preconditioners and provide a formal convergence proof based on a framework originally used for on-line learning. Inspired by recent popular adaptive per-feature algorithms, we propose a specific preconditioner based on the second moment of the gradient. The sufficient convergence conditions motivate a critical adaptation of the per-feature updates in order to ensure convergence. We show empirical results for the minimization of convex and non-convex cost functions, in the context of neural network training. The method compares favorably with respect to current, first order, stochastic optimization methods.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper talks about a method to combine preconditioning at the per feature level and Nesterov-like acceleration for SGD optimization. The explanation of the method in Section 3 should be self-contained. The main result, computational context, etc., are poorly described, so that it would not be easily understandable to a non-expert. What was the reason for the choice of the mini batch size of 128. I would guess that you would actually see interesting differences for the method by varying this parameter. How does this compare with the FLAG method of Chen et al from AISTATS, which is motivated by similar issues and addresses similar concerns, obtaining stronger results as far as I can tell? The figures and captions and inserts are extremely hard to read, so much so that I have to trust it when the authors tell me that their results are better. The empirical evaluation for "convex problems" is for LS regression. Hmm. Is there not a better convex problem that can be used to illustrate the strength and weaknesses of the method. If not, why don*t you compare to a state-of-the-art least squares solver. For the empirical results, what looks particularly interesting is some tradeoffs, e.g, a slower initial convergence, that are shown. Given the limited scope of the empirical evaluations, it*s difficult to tell whether there is much to argue for the method. But those tradeoffs are seen in other contexts, e.g., with subsampled second order methods, and it would be good to understand those tradeoffs, since that might point to where and if a methods such as this is useful. The conclusions in the conclusion are overly broad.
assistant
 The paper talks about a method to combine preconditioning at the per feature level and Nesterov-like acceleration for SGD optimization. The explanation of the method in Section 3 should be self-contained. The main result, computational context, etc., are poorly described, so that it would not be easily understandable to a non-expert. What was the reason for the choice of the mini batch size of 128. I would guess that you would actually see interesting differences for the method by varying this parameter. How does this compare with the FLAG method of Chen et al from AISTATS, which is motivated by similar issues and addresses similar concerns, obtaining stronger results as far as I can tell? The figures and captions and inserts are extremely hard to read, so much so that I have to trust it when the authors tell me that their results are better. The empirical evaluation for "convex problems" is for LS regression. Hmm. Is there not a better convex problem that can be used to illustrate the strength and weaknesses of the method. If not, why don*t you compare to a state-of-the-art least squares solver. For the empirical results, what looks particularly interesting is some tradeoffs, e.g, a slower initial convergence, that are shown. Given the limited scope of the empirical evaluations, it*s difficult to tell whether there is much to argue for the method. But those tradeoffs are seen in other contexts, e.g., with subsampled second order methods, and it would be good to understand those tradeoffs, since that might point to where and if a methods such as this is useful. The conclusions in the conclusion are overly broad.