Despite many second-order methods have been proposed to train neural networks, most of the results were done on smaller single layer fully connected networks, so we still cannot conclude whether it's useful in training deep convolutional networks. In this study, we conduct extensive experiments to answer the question "whether second-order method is useful for deep learning?". In our analysis, we find out although currently second-order methods are too slow to be applied in practice, it can reduce training loss in fewer number of iterations compared with SGD. In addition, we have the following interesting findings: (1) When using a large batch size, inexact-Newton methods will converge much faster than SGD. Therefore inexact-Newton method could be a better choice in distributed training of deep networks. (2) Quasi-newton methods are competitive with SGD even when using ReLu activation function (which has no curvature) on residual networks. However, current methods are too sensitive to parameters and not easy to tune for different settings. Therefore, quasi-newton methods with more selfadjusting mechanisms might be more useful than SGD in training deeper networks.
The paper conducts an empirical study on 2nd-order algorithms for deep learning, in particular on CNNs to answer the question whether 2nd-order methods are useful for deep learning.  More modestly and realistically, the authors compared stochastic Newton method (SHG) and stochastic Quasi- Newton method (SR1, SQN) with stochastic gradient method (SGD).  The activation function ReLu is known to be singular at 0, which may lead to poor curvature information, but the authors gave a good numerical comparison between the performances of 2nd-order methods with ReLu and the smooth function, Tanh.  The paper presented a reasonably good overview of existing 2nd-order methods, with clear numerical examples and reasonably well written.
The paper presents several interesting empirical findings, which will no doubt lead to follow up work. However, there are also a few critical issues that may undermine their claims, and that need to be addressed before we can really answer the original question of whether 2nd-order methods are useful for deep learning. 
1. There is no complexity comparison, e.g. what is the complexity for a single step of different method.
2. Relatedly, the paper reports the performance over epochs, but it is not clear what "per epoch" means for 2nd-order methods.  In particular, it seems to me that they did not count the inner CG iterations, and it is known that this is crucial in running time and important for quality.  If so, then the comparison between 1st-order and 2nd-order methods are not fair or incomplete.
3. The results on 2nd-order methods behave similarly to 1st-order methods, which makes me wonder how many CG iterations they used for 2nd-order method in their experiment, and also the details of the data.  In particular, are they looking at parameter/hyperparameter settings for which 2nd-order methods aren't really necessary.
4. In deep learning setting, the training objective is non-convex, which means the Hessian can be non-PSD.  It is not clear how the stochastic inexact-Newton method mentioned in Section 2.1 could work.  Details on implementations of 2nd-order methods are important here.
5. For 2nd-order methods, the author used line search to tune the step size.  It is not clear in the line search, the author used the whole training objective or batch loss.  Assuming using the batch loss, I suspect the training curve will be very noisy (depending on how large the batch size is).  But the paper only show the average training curves, which might be misleading.
Here are other points.
1. There is no figure showing training/ test accuracy.  Aside from being interested in test error, it is also of interest to see how 2nd order methods are similar/different than 1st order methods on training versus test.
2. Since it is a comparison paper, it only compares three 2nd-order methods with SGD.  The choices made were reasonable, but 2nd-order methods are not as trivial to implement as SGD, and it isn't clear whether they have really "spanned the space" of second order methods
3. In the paper, the settings of LeNet, AlexNet are different with those in the original paper.  The authors did not give a reason.
4. The quality of figures is not good.
5. The setting of optimization is not clear, e.g. the learning rate of SGD, the parameter of backtrack line search.  It's hard to reproduce results when these are not described.