Although mathematical expressions (MEs) recognition have achieved great progress, the development of MEs recognition in real scenes is still unsatisfactory. Inspired by the recent work of neutral network, this paper proposes a novel two-stage approach which takes a printed mathematical expression image as input and generates LaTeX sequence as output. In the first stage, this method locates and recognizes the math symbols of input image by object detection algorithm. In the second stage, it translates math symbols with position information into LaTeX sequences by seq2seq model equipped with attention mechanism. In particular, the detection of mathematical symbols and the structural analysis of mathematical formulas are carried out separately in two steps, which effectively improves the recognition accuracy and enhances the generalization ability. The experiment demonstrates that the two-stage method significantly outperforms the end-to-end method. Especially, the ExpRate(expression recognition rate) of our model is 74.1%, 20.3 percentage points higher than that of the end-to-end model on the test data that doesn't come from the same source as training data.
This paper considers the problem of converting an image of a math expression into LaTeX.  They note that while the model proposed in Deng et al works well on the IM2LATEX-100K dataset, is doesn't generalize well to equations in real-world settings that you'd have in a photograph or a scan of an equation.  They propose an approach that breaks the problem into two steps.  In the first step they detect all the characters in the image, identifying the  character type and bounding box for each.  In the second step they use an encoder/decoder (LSTM/LSTM with attention) model to translate this sequence of character encodings into a LaTeX sequence.  They create a new dataset of LaTeX equations rendered on backgrounds sampled from real photographs of books and papers.  They split this into a training and "homologous" test set.   They find performance of their two stage model is a bit better than Deng's model on this test set.  They then create a "non-homologous" test set, in which the test equations are rendered on a new set of backgrounds, unseen in training.  On this test set, the model in the paper performs essentially the same as on the homologous dataset, while the Deng model performs substantially worse.  The conclusion is that the two stage approach creates a model that is much more robust to the appearance of the equation.
I think the main takeaway from this paper is that there can be disadvantages to an end-to-end approach.  To achieve their improvement, authors used their insight that there is an intermediate representation that summarizes all relevant information (the sequence of characters and positions), together with the ability to generate a new training set automatically to learn this intermediate representation.
I think this is an interesting case study in applied machine learning, but I don't think it will be of enough general use or interest to the ICLR community to merit acceptance.
As an applications paper, I think there are several aspects that can be improved.  Here are some specific questions and comments that may help a further iteration of this paper:
- You have examples of ME images from the real world, but you don't have any examples of your artificial "real world" equations, overlayed on your sampled backgrounds.  Those would be helpful. Are any other modifications done to the equation to simulate a real-life picture or scan, such as color or darkness distortions, angles, etc?
- How does your proposed model perform on the original IM2LATEX-100K problem?
- How well does the encoder-decoder model do an a gold encoding of the input?  It would be nice to separate the errors into translation errors vs object detection errors.
- Can you give examples of scenarios where your model got things right and Deng's model did not, and vice versa?  Is there any interpretation to why each model does better or worse for various example?  I imagine the model may have more difficulty with equations where one needs to refer to characters that are on the left of the image quite late in the LaTeX expression.  For example, fractions with long numerator expressions and and cases environments.
- Did you try or consider using a soft encoding of the character identity, instead of a one-hot?  Perhaps there are context clues that the encoder/decoder model could use to disambiguate between a 1 and an l, for example.