Generative Shape Models: Joint Text Recognition and Segmentation with Very Little Training Data
Abstract: We demonstrate that a generative model for object shapes can achieve state of the art results on challenging scene text recognition tasks, and with orders of magnitude fewer training images than required for competing discriminative methods. In addition to transcribing text from challenging images, our method performs fine-grained instance segmentation of characters. We show that our model is more robust to both affine transformations and non-affine deformations compared to previous approaches.
This aim of this paper is to recognize non-handwritten text in images. The model is generative, and inference proceeds using belief propagation on a factor graph for both finding characters in images and parsing words. Word parsing is based on a combination of bigram and trigram language models. Learning proceeds by learning structural models of characters for fonts, and then by greedily selecting fonts to be modelled. Experiments are performed on two font datasets, where the experimental results are incrementally better than OCR, though with a greatly reduced amount of training examples.
First, the experimental results seem fine, albeit a incremental. They slightly outperform competing methods. Although it's true the authors achieve good results with less training data than competing methods, their method is also expensive and it doesn't seem easy to make use of more training data anyway.  Next, this does not seem to be an actual generative model, but a probabilistic model. Because of how the character recognition is setup (a loopy factor graph), and how parsing is performed, it doesn't seem like one could actually generate text with this model. In other words, sampling would seems prohibitively expensive. It would be more fair to say this is a probabilistic model, but not a generative one for which sampled texts could be drawn. The authors state that the framework can generate segmentation boundaries (line 436); can it really? How do you sample from the factor graphs? There are these big normalizing constants in there, and the model is not a DAG, so sampling doesn't seem easy. How are boundary segmentations generated? It would be helpful to see samples from the model.  The inference step seems a bit odd. The authors build a minimum-spanning-tree over the edges of the factor graph. How is this done? Factor graphs don't have weights, so how is this spanning tree being created? I found this unclear. How necessary is the MST? How many edges does the loopy graph have, typically? Are the potentials constructed in such a way that message passing is linear in clique size, or is it more expensive? These details are missing in the paper.  Also, the authors state that they have "long constraints" in their factor graph for inferring characters, to help model the structure of characters. These constraints of a parameter (\gamma), and it's unclear how this value was set, and whether its value is important. Is the model/inference sensitive to the choice of parameter value?  Another modelling problem is the need to model fonts individually. Why is this necessary? Could there not be a richer model of what characters in general look like? What happens when one tries to perform inference on a font that the system hasn't seen (and hasn't seen anything similar)? Would the system be able to state that it's highly uncertain about the parse it produces for this unseen font?  Lastly, there is a factor graph for detecting characters which produces marginals, and another factor graph for word parsing. Why not combine the two into one factor graph to do joint character detection and parsing? Why separate the two stages?  Overall, the inference and learning seem ad-hoc, and a lot of details are missing. Inference in a large factor graph is challenging, but from the setup of the paper, there is some machinery in the framework that is complicated and seems unnecessary (eg, dividing the factor graph into two stages). Also, the modelling of characters seems brittle and that in-plane word rotations would cause the system to fail; although rotations could be built into the system in theory, it seems impractical here. The experimental results are incremental. Lastly, the margins are way too small, so there is a lot more content here than would normally be allowed.