The learning rate warmup heuristic achieves remarkable success in stabilizing training, accelerating convergence and improving generalization for adaptive stochastic optimization algorithms like RMSprop and Adam. Pursuing the theory behind warmup, we identify a problem of the adaptive learning rate -its variance is problematically large in the early stage, and presume warmup works as a variance reduction technique. We provide both empirical and theoretical evidence to verify our hypothesis. We further propose Rectified Adam (RAdam), a novel variant of Adam, by introducing a term to rectify the variance of the adaptive learning rate. Experimental results on image classification, language modeling, and neural machine translation verify our intuition and demonstrate the efficacy and robustness of RAdam.
I haven't worked in this area before, and my knowledge to the topic of designing optimizers for improving stochastic gradient descent is limited to the level of advanced ML courses at graduate school. Nevertheless, below I try my best to evaluate the technicality of this paper
====================
In this work the authors studied the variance issue of the adaptive learning rate and aim to justify the warm-start heuristic. They also demonstrate that  the convergence issue of many of the stochastic gradient descent algorithms is due to large variance induced by the adaptive learning rate in the early stage of training. To tackle this issue, they proposed a variant of ADAM, which is known as rectified ADAM, whose learning rate not only takes the momentum into the account, but it also adapts to the variance of the previous gradient updates.  
In the first part of the paper, the authors analyzed the variance issue exists in the existing ADAM algorithm, such that with limited samples in the early stage of training, the variance of the adaptive learning rate becomes rather large and it induces high variance to the gradient update to ADAM. In general I found this theoretical justification on the observation of variance issue in ADAM sound, and quite intuitive. In the second part, they proposed the algorithm, namely rectified ADAM, where the difference here to take the second moment  of the gradient into account when updating the adaptive learning rate. They showed that the variance of the adaptive learning rate with such rectification is more numerically stable (especially when variance is intractable in vanilla ADAM), and under some regularity assumption it decreases in the order or O(1/\rho_t).
In extensive numerical studies of supervised learning, the authors showed that RADAM achieves a better accuracy than ADAM (although in Table 1, I am a bit puzzled why the best accuracy is indeed from SGD, if so, what's the point of all adaptive learning rates, is that because SGD requires extensive lr tuning?) Because the superiority in accuracy they also showed that RADAM manages to have more stable training and achieves lower training loss, which is quite interesting.