Optimization algorithms for training deep models not only affects the convergence rate and stability of the training process, but are also highly related to the generalization performance of trained models. While adaptive algorithms, such as Adam and RMSprop, have shown better optimization performance than stochastic gradient descent (SGD) in many scenarios, they often lead to worse generalization performance than SGD, when used for training deep neural networks (DNNs). In this work, we identify two problems regarding the direction and step size for updating the weight vectors of hidden units, which may degrade the generalization performance of Adam. As a solution, we propose the normalized direction-preserving Adam (ND-Adam) algorithm, which controls the update direction and step size more precisely, and thus bridges the generalization gap between Adam and SGD. Following a similar rationale, we further improve the generalization performance in classification tasks by regularizing the softmax logits. By bridging the gap between SGD and Adam, we also shed some light on why certain optimization algorithms generalize better than others.
Method:
The paper is missing analysis of some important related works such as
"Beyond convexity: Stochastic quasi-convex optimization" by E. Hazan et al. (2015) 
where Stochastic Normalized Gradient Descent (SNGD) was proposed. 
Then, normalized gradient versions of AdaGrad and Adam were proposed in 
"Normalized Gradient with Adaptive Stepsize Method for Deep
Neural Network Training" by A. W. Yu et al. (2017).
Another work which I find to be relevant is 
"Follow the Signs for Robust Stochastic Optimization" by L. Balles and P. Hennig (2017).
From my personal experiments, restricting w_i to have L2 norm of 1, i.e., to be +-1 
leads to worse generalization. One reason for this is that weight decay is not 
really functioning since it cannot move w_i to 0 or make its amplitude any smaller. 
Please correct me if I misunderstand something here. 
The presence of +-1 weights moves us to the area of low-precision NNs, 
or more specifically, NNs with binary / binarized weights as in 
"BinaryConnect: Training Deep Neural Networks with
binary weights during propagations" by M. Courbariaux et al. (2015)
and 
"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1" by M. Courbariaux et al. (2016). 
Regarding
"Moreover, the magnitude of each update does not depend on themagnitude of the gradient. Thus, ND-Adam is more robust to improper initialization, and vanishing or exploding gradients."
If the magnitude of each update does not depend on the magnitude of the gradient, then the algorithm heavily depends on the learning rate. Otherwise, it does not have any means to approach the optimum in a reasonable number of steps *when* it is initialized very / unreasonably far from it. The claim of your second sentence is not supported by the paper. 
Evaluation:
I am not confident that the presented experimental validation is fair. First, the original WRN paper and many other papers with ResNets used weight decay of 0.0005 and not 0.001 or 0.002 as used for SGD in this paper. It is unclear why this setting was changed. One could just use \alpha_0 = 0.05 and \lambda = 0.0005.
Then, I don't see why the authors use WRN-22-7.5 which is different from WRN-28-10 which was suggested in the original study and used in several follow-up works. The difference between WRN-22-7.5 and WRN-28-10 is unlikely to be significant, 
the former might have about only 2 times less parameters which should barely change the final validation errors. However, the use of WRN-22-7.5 makes it impossible to easily compare the presented results to the results of Zagoruyko who had 3.8\% with WRN-28-10. I believe that the use of the setup of Zagoruyko for WRN-22-7.5 would allow to get much better results than 4.5\% and 4.49\% shown for SGD and likely better 4.14\% shown for ND-Adam. I note that the use of WRN-22-7.5 is unlikely to be due to the used hardware because later in paper the authors refer to WRN-34-7.5.
My intuition is that the proposed ND-Adam moves the algorithm back to SGD but with potentially harmful constraints of w_i=+-1. Even the values of \alpha^v_0 found for ND-Adam (e.g., \alpha^v_0=0.05 in Figure 1B) are in line of what would be optimal values of \alpha_0 for SGD. 
I find it uncomfortable that BN-Softmax is introduced here to support the use of an optimization algorithm, moreover, that the values of \gamma_c are different for CIFAR-10 and CIFAR-100. I wonder if the proposed values are optimal (and therefore selected) for all three tested algorithms  or only for Adam-ND. I expect that hyperparameters of SGD and Adam would also need to be revised to account for BN-Softmax.