Hessian-free Optimization for Learning Deep Multidimensional Recurrent Neural Networks
Multidimensional recurrent neural networks (MDRNNs) have shown a remarkable performance in the area of speech and handwriting recognition. The performance of an MDRNN is improved by further increasing its depth, and the difficulty of learning the deeper network is overcome by using Hessian-free (HF) optimization. Given that connectionist temporal classification (CTC) is utilized as an objective of learning an MDRNN for sequence labeling, the non-convexity of CTC poses a problem when applying HF to the network. As a solution, a convex approximation of CTC is formulated and its relationship with the EM algorithm and the Fisher information matrix is discussed. An MDRNN up to a depth of 15 layers is successfully trained using HF, resulting in an improved performance for sequence labeling.
This paper proposes using HF for learning a MDRNN with a CTC loss on top. The contributions of this work are to show that a loss that is non-convex (CTC) can still be used within Hessian Free by means of decomposing the loss using similar tricks than Gauss Newton matrices (which are applied already in HF methods for NNs). The authors show an elegant connection between their effort to make CTC suitable for HF, and an EM procedure.The authors of this paper showed an effective way to train a very complex model (MDRNN), with a non-linear cost (CTC) using second order information (HF). The main theoretical contribution is to make CTC compatible with HF by proposing a GGN that is PSD, a requirement needed for CG within HF. Their contribution goes beyond that as they show interesting links with EM, and provide empirical evidence that the method is working.I, however, have a few concerns regarding the experiments presented in this work:-As the authors discuss, TIMIT is mostly an overfitting game and not so much and underfitting / optimization game. What was the training cost achieved by both methods?-There are other baselines that may be more practical (i.e. less implementation details that may go wrong), such as Adagrad, RMSProp, or Adam. Could the authors provide such baselines? -It would be useful to add some prior art in the tables for comparison.As a final note, there are quite a lot of grammatical and minor mistakes in the text. I'd suggest a deep pass over those before publishing this work.