Second-order methods for neural network optimization have several advantages over methods based on first-order gradient descent, including better scaling to large mini-batch sizes and fewer updates needed for convergence. But they are rarely applied to deep learning in practice because of high computational cost and the need for model-dependent algorithmic variations. We introduce a variant of the Hessian-free method that leverages a block-diagonal approximation of the generalized Gauss-Newton matrix. Our method computes the curvature approximation matrix only for pairs of parameters from the same layer or block of the neural network and performs conjugate gradient updates independently for each block. Experiments on deep autoencoders, deep convolutional networks, and multilayer LSTMs demonstrate better convergence and generalization compared to the original Hessian-free approach and the Adam method.
Summary: 
The paper considers second-order optimization methods for training of neural networks.
In particular, the contribution of the paper is a Hessian-free method that works on blocks of parameters (this is a user defined splitting of the parameters in blocks, e.g., parameters of each layer is one block, or parameters in several layers could constitute a block). 
This results into a block-diagonal approximation to the curvature matrix, in order to improve Hessian-free convergence properties: in the latter, a single step might require many CG steps, so the benefit from using second-order information is not apparent.
This is mainly an experimental work, where the authors show the merits of their approach on deep autoencoders, convolutional networks and LSTMs: results show favourable performance compared to the original Hessian-free approach and the Adam method.
Originality: 
The paper is based on the works of Collobert (2004) and Le Roux et al. (2008), as well as the work of Martens: the twist is that each layer of the neural network is considered a parameter block, so that gradient interactions among weights in a single layer are more useful than those between weights in different layers. This increases the separability of the problem and reduces the complexity. 
Importance: 
Understanding the difference between first- and second-order methods for NN training is an important topic. Using second-order methods could be considered at its infancy, compared to the wide variety of first-order methods. Having new results on second-order methods with interesting results would definitely attract some attention at the conference. 
Presentation/Clarity: 
The paper is well structured and well written. The authors clearly place their work w.r.t. state of the art and previous works, so that it is clear what is new and what is known.
Comments:
1. It is not clear why the deficiency of first-order methods on training NNs with big batches motivates us to turn into second-order methods. Is there a reasoning for this statement? Or is it just because second-order methods are kind-of the only other alternative we have?
2. Assuming we can perform a second-order method, like Newton's method, on a deep NN. Since originally Newton's method was designed to find solutions that have gradient equal to zero, and since NNs have saddle points (probably many more than local minima), even if we could perfectly perform second-order Newton motions, there is no guarantee whether we converge to a local minimum or a saddle point. However, since we perform Newton's method approximately in practice, this might help escaping saddle points. Any comment on this aspect (I'm not aware whether this is already commented in Schraudolph 2002, where the Gauss-Newton matrix was proposed instead of the Hessian)?