Stochastic gradient descent (SGD) and Adam are commonly used to optimize deep neural networks, but choosing one usually means making tradeoffs between speed, accuracy and stability. Here we present an intuition for why the tradeoffs exist as well as a method for unifying the two in a continuous way. This makes it possible to control the way models are trained in much greater detail. We show that for default parameters, the new algorithm equals or outperforms SGD and Adam across a range of models for image classification tasks and outperforms SGD for language modeling tasks.
This paper proposes a new algorithm which brings closer SGD and Adam while also incorporating new corrections to improve behavior of the optimizer in contexts where there is very small or very large eigen values.
Decision
I vote for weak rejection because the core modification proposed to Adam is minor and is mostly supported by intuition and preliminary experiments.
Justification
There is many modifications proposed, but most are secondary corrections for stability, such as the warm-up schedule with the redefinition of beta_2. These modifications could as well be incorporated in Adam without the core modification that is the smoothing presented in section 3. These additional modifications also make it difficult to measure the importance of the core contribution. Without getting rid of them, an ablation study on toy problems (even synthetic data) would be necessary for a better understanding.
In section 3.1, the temporal definition of beta_2t is integrating a warm-up. While the reason for doing so is supported in introduction of section 3, the effect of this modification should be weighted against no warm-up, and also compared with its effect on Adam.
There is an error in algorithm 1. The last element of the last line (Perform the update) should be \alpha (1 - \eta) m_t/d_t. The code in Appendix corroborates this correction. Minor related note, the use of d_t to define the denominator of what d_i represents in section 3 is very confusing. I would suggest to use the ratio notation of d_i from the equations in the algorithm for coherence.
If we get pass the warm-up scheduling, by massaging the equation we get that the algorithm is different from Adam on 2 points, 1) the bias are not corrected and 2) the denominator sqrt(v) + epsilon is replaced by sqrt(v) + sqrt(mean(v) + epsilon^2). I have difficulty convincing myself that smoothing by the average is solving the issues raised in the paper and there is no experiments to study its effect directly.
The experiments are on 3 datasets, but only the computer vision ones are run on multiple architectures. Caption of figure 1 explains that each model is trained 3 times, but the source of variation between each run is not described. Are the models initialized differently? In any case, there is an overlap for 3 of the 5 models between SGD and SoftAdam which makes the comparison rather unconvincing. There is no standard deviation for Adam, and none on Penn Treebank dataset and IWSLT. For a better comparison, all hyper-parameters of the algorithms should be optimized for each run. I understand that SoftAdam is meant to be close to both SGD and Adam, but using the same hyper-parameters may induces misleading results by favoring some (model, optimizer) combination nevertheless.
Minor comments
In section 2, second paragraph, the term 'mini-batch' should be used instead of 'batch'.
In section 2, last sentence, the betas should have no t.
In section 2.1, fourth equation (unnumbered), the eigen vector xi_i is presented as a vector and then used as a scalar. Notation should be uniformed.
In Section 2.1 around equation (2), the use of i and j is incoherent.
In Section 3:
- Overall, this understanding *of* has
- we consider the *an* update