Joint Line Segmentation and Transcription for End-to-End Handwritten Paragraph Recognition
Offline handwriting recognition systems require cropped text line images for both training and recognition. On the one hand, the annotation of position and transcript at line level is costly to obtain. On the other hand, automatic line segmentation algorithms are prone to errors, compromising the subsequent recognition. In this paper, we propose a modification of the popular and efficient Multi-Dimensional Long Short-Term Memory Recurrent Neural Networks (MDLSTM-RNNs) to enable end-to-end processing of handwritten paragraphs. More particularly, we replace the collapse layer transforming the two-dimensional representation into a sequence of predictions by a recurrent version which can select one line at a time. In the proposed model, a neural network performs a kind of implicit line segmentation by computing attention weights on the image representation. The experiments on paragraphs of Rimes and IAM databases yield results that are competitive with those of networks trained at line level, and constitute a significant step towards end-to-end transcription of full documents.
The authors investigate the introduction of an attention mechanism into an offline handwriting recognition system pointed at unsegmented paragraphs of handwritten text. MD-LSTMs are used for feature extraction, attention computation, and (in some cases) decoding. Interestingly, learned attention is shown to outperform ground truth segmentation in the authors' system. Character error rates on standard datasets appear to be quite impressive. While the system's word error rates do not break state of the art, they are quite competitive despite ignoring both ground truth segmentation and employing a relatively simplistic language model. 
Positives: -The idea of applying technique to generalize single-line handwriting recognition to multi-line is both an elegant idea and a natural next step. It is unsurprising that this is one of at least two works (mentioned by the authors) attempting similar techniques (not that this is relevant, but I consider this to be the more mature/developed work of the two). -It is impressive (as is often the case these days) that such a system works at all, let alone that it works as well as it does. -The technical exposition provided by the authors is sufficient, and is generally easy to follow. The colorized attention figure is particularly gratifying.  Areas for improvement: -The comparison to established benchmarks raises more questions than it provides answers. As the authors note, this comparison is apples-to-oranges. The reference models exploit more complex language models and make use of ground truth segmentation. The latter is unavoidable due to the fundamental nature of the problem considered, but the gap in language models is quite unnecessary. Discussion of this comparison also feels underdeveloped and unenlightening, and I am confident the authors could do better. -The authors do not adequately discuss the significance/origin of the learned attention outperforming the ground truth. Is it because of noise in the annotations? Leveraging information contained in neighboring lines? This is particularly important, as the models compared to in Sec 5.4 all make use of this ground truth. -Finally, while the paper is well organized and easy to follow, the quality of the writing leaves something to be desired. There are far too many typos, grammatical errors, and shifting tenses. Furthermore, particularly when comparing against other work, the occasionally competitive tone undermines the authors' credibility. Their results stand up quite well, even in a neutral light.