Several stochastic optimization algorithms are currently available. In most cases, selecting the best optimizer for a given problem is not an easy task. Therefore, instead of looking for yet another absolute best optimizer, accelerating existing ones according to the context might prove more effective. This paper presents a simple and intuitive technique to accelerate first-order optimization algorithms. When applied to first-order optimization algorithms, it converges much more quickly and achieves lower function/loss values when compared to traditional algorithms. The proposed solution modifies the update rule, based on the variation of the direction of the gradient during training. Several tests were conducted with SGD, AdaGrad, Adam and AMSGrad on three public datasets. Results clearly show that the proposed technique, has the potential to improve the performance of existing optimization algorithms.
Summary:
This paper presents an adaptive method which can be used alongside existing accelerated gradient methods. The paper is difficult to read due to mistakes and poorly defined mathematical notation. I believe that the paper is missing reference to related methods. The theoretical analysis in the paper is difficult to follow and provides little insight into the benefits of the proposed approach.
Overview:
There are many mistakes throughout the paper which have made it difficult to read.
Overall, I felt that this paper was missing a discussion of the effect of stochasticity on the proposed method. The issue with measuring the variation in the gradient direction is that in regimes where the gradient noise is dominating the signal the gradient direction at each time step is poorly correlated with overall optimization progress --- thus it seems intuitively ineffective to rely on the gradient direction to adjust the algorithm.
1)  At the bottom of page 2, the authors write "knowing that we do not have any knowledge of what this function looks like". While minor, I would point out that we are able to compute local statistics of the function and so certainly we have _some_ knowledge.
2) The authors claim that no techniques exist which use the variation of the direction of the gradient. One such example is in [1] which uses (in one case) the variation of the gradient direction to determine and appropriate time to restart the momentum computation. 
3) In section 3.2, the Adam moment computation is missing a "diag". Assuming that AMRSGrad is AMSGrad (mistyped), then this term is incorrect and matches Adam.
4) There are many mistakes in the Algorithm 1 box.
- The wrong $F$ is used in the input (should be $\mathcal{F}$).
- The algorithm takes as input a sequence of functions ($\phi, \psi$) which are not used.
- Within the if statement, $gm_t = g_t + m_t$. I believe this should be an $m_{t-1}$. It is not clear what the vector $gm$ is exactly, and then $\dot{g}$ is used afterwards which is also not defined.
- The algorithm checks for $|m_{t-1} - g_t| > S$ while the text uses $|g_{t-1} - g_t| > S$.
5) The first line of section 3.3 is quite worrying: "We assume that if we are able to prove that modifying one optimizer with the proposed method does not alter its convergence, then the same applies for the other optimizers". This seems like a dangerous assumption to make and should at the very least be carefully verified empirically. Following this, I am not sure what the authors mean by "deterministic" and "non-deterministic" methods.
6) I do not understand the claim above Theorem 2 that $\nabla f(x_{T-1}) = k \nabla f(x_T)$. Under what conditions does this hold and how is $k$ computed? If I understand correctly, the bound provided in Theorem 2 is worse than that given for gradient descent. Moreover, the bound does not depend on the hyperparameter $S$ introduced in Algorithm 1 and provides limited insights into the method. I could not find a proof of Theorem 2 in the paper or appendix.
7) There are serious flaws with the experimental evaluation in this paper.
a) There is no tuning over hyperparameter settings for any of the optimizers.
b) The basic problems are very limited, even for toy problems. The 1D deterministic quadratic tells us very little about the performance of the optimizer. And the 1D cubic problem is particularly confusing. Unless I am mistaken, the gradient will always have the same sign (3x^2) and thus the acceleration condition will never be triggered.
c) I believe that Figure 2 explores stochastic optimization problems which as discussed at top is a crucial evaluation. Unfortunately, due to lack of parameter tuning it is difficult to infer much about the comparison between the methods.
d) Figure 4 compares performance variation over changing the threshold. The y-axis scale across each plot changes making the comparison unnecessarily difficult --- the scale should be the same.
Minor:
- TYPO Line 2, "minimize ---,"
- End of intro, MNIST and CIFAR not cited while IMDB is. Citation uses citet not citep.
- Bottom of page 2, ""
- Top of section 3.2, "The pseudo code of our the method"
References:
[1] Adaptive Restart for Accelerated Gradient Schemes, Brendan O'Donoghue and Emmanuel Candes