Although adaptive algorithms have achieved significant success in training deep neural networks with faster training speed, they tend to have poor generalization performance compared to SGD WITH MOMENTUM(SGDM). One of the state-of-the-art algorithms, PADAM, is proposed to close the generalization gap of adaptive methods while lacking an internal explanation. This work proposes a general framework, in which we use an explicit function Φ(·) as an adjustment to the actual step size, and present a more adaptive specific form ADAPLUS(ADA+). Based on this framework, we analyze various behaviors brought by different types of Φ(·), such as a constant function in SGDM, a linear function in ADAM, a concave function in PADAM and a concave function with offset term in ADAPLUS. Empirically, we conduct experiments on classic benchmarks both in CNN and RNN architectures and achieve better performance(even than SGDM). The code is anonymously provided in https: //anonfiles.com/daV7Ed6enb/AdaPlus_zip.
This paper proposes a new variation on adaptive learning rate algorithm that builds on a prior work Padam, which is also a concurrent submission to this conference. The method is empirically validated through through various domains and neural networks architecture. Though the empirical results are extensive, I am leaning towards reject because (1) The method is a very small variation; no insight is provided for why. (2) The theory is not very useful in justifying the method. (3) The empirical results are weak.
(1) This work builds on the prior work Padam, which is not a well justified algorithm. Same criticisms hence can be made to this paper. I don't see any strong arguments from the paper that well justifies the methodology either.
The problems are listed below:
Here's what I can find of the authors' interpretation of the prior work Padam: 
"The internal cause is that a concave function is applied rather than the linear function in ADAM.
Once ε is extremely small and |g| ∈ (0, ε), the mapping value of Φ(·) would be much
larger in PADAM than in ADAM; therefore, PADAM can adapt to larger learning rate α,
thus flexibly adapting to the variable learning rate scheme."
(i) I don't think phrasing the cause as a result of concavity gives arise to any new insights. Simply, what this is trying to say is that Φ(·) is large when |g| is small, that's why PADAM can adapt to large learning rate. 
Building on Padam, the paper further justifies the proposed method by:
" This form of Φ(·) not only directly inherits advantages of PADAM,
as is depicted in Figure 1, but also makes a better guarantee for larger learning rates. The
offset ∆ makes sure that Φ(·) can altogether avoid the extreme situation. Even when
|g| → 0, a more extensive learning rate αt is allowed. "
(ii) The paper inherits a serious problem from Padam, that is to assume large learning rate is important for learning rate decay. Padam didn't explain this, nor does this paper. So the justification is very weak. 
(iii) The introduction of the offset is not novel. Just as the author noted, this is almost the same term epsilon in original updates. It's not rare that one also tunes epsilon for some optimization problems.
(2) Continuing the last point, the theoretical analysis focus on the convergence proof of the algorithm. The proof is not new. Theory also is not useful in justifying the method.
(3) The method seems not to be able to beat the previous baseline Padam, which makes it questionable as a practical algorithm.