Stochastic methods with coordinate-wise adaptive stepsize (such as RMSprop and Adam) have been widely used in training deep neural networks. Despite their fast convergence, they can generalize worse than stochastic gradient descent. In this paper, by revisiting the design of Adagrad, we propose to split the network parameters into blocks, and use a blockwise adaptive stepsize. Intuitively, blockwise adaptivity is less aggressive than adaptivity to individual coordinates, and can have a better balance between adaptivity and generalization. We show theoretically that the proposed blockwise adaptive gradient descent has comparable regret in online convex learning and convergence rate for optimizing nonconvex objective as its counterpart with coordinate-wise adaptive stepsize, but is better up to some constant. We also study its uniform stability and show that blockwise adaptivity can lead to lower generalization error than coordinate-wise adaptivity. Experimental results show that blockwise adaptive gradient descent converges faster and improves generalization performance over Nesterov's accelerated gradient and Adam.
The paper proposes adaptive gradient approaches where the step-size is not determined on the per-coordinate basis but rather for blocks of coordinates. Theoretical results are presented in terms of regret in online convex optimization, regarding convergence in non-convex optimization,  and with respect to uniform stability and generalization. These indicate that under certain conditions adaptivity at the block level could outperform coordinate-wise adaptivity. The approach is evaluated against alternatives on simulated and real-world problems.
The paper considers an important topic, which has been the object of many related studies. Though the proposed approach is interesting, there are several issues with the present manuscript that would warrant significant revision.
- Though the discussion in section 3 aims at motivating the use of block-wise adaptivity, it is quite confusing. Indeed the problem considered in that section is different from the setup eventually considered.  Moreover the paper claims that it is more general than the many previous work on layer-wise adaptation. However, the example considered here does consider layers. 
- The paper organization could be improved significantly. BAG is presented followed by a regret analysis in convex optimization. Then BAGM is presented followed by the theory on non-convex optimization. It might be more effective to first present the algorithms BAG and BAGM and then have a theory section.  It would also be good to emphasize that the regret analysis is solely for convex optimization, which makes it much less relevant in the context of deep learning.
- The claimed superiority of block-wise adaptivity in the theory relies heavily on an assumption relying on tightness and closeness of  upper bounds on gradient magnitude /  gradient second moment in each block.  As this assumption is crucial to the results, its validity in practice should be more thoroughly investigated, beyond the small study relegated in the appendix  B. It would also be important to assess what happens when the assumption breaks down. 
- The aforementioned issue is somewhat also related to the choice of design for the blocks. Some choices might help, some might hurt based on the unknown structure of the data. This is barely touched upon in section 5.1. as it is noted that the more sever the mismatch the worse the results. 
- The empirical evaluation needs to be more comprehensive in terms of comparison methods. Among others it would be important to assess performance against previously proposed layer wise stepwise approaches mentioned in the introduction. 
- These related approaches should also be discussed more deeply and contrasted against qualitatively.
- The empirical results are not substantially superior. It is also quite disappointing to see that the training, testing and generalization curves on Figure 2 are quite similar with NAG and Adam and do not exhibit less instability etc.