  Reading of mathematical expression or equation in the document images is very
challenging due to the large variability of mathematical symbols and
expressions. In this paper, we pose reading of mathematical equation as a task
of generation of the textual description which interprets the internal meaning
of this equation. Inspired by the natural image captioning problem in computer
vision, we present a mathematical equation description (MED) model, a novel
end-to-end trainable deep neural network based approach that learns to generate
a textual description for reading mathematical equation images. Our MED model
consists of a convolution neural network as an encoder that extracts features
of input mathematical equation images and a recurrent neural network with
attention mechanism which generates description related to the input
mathematical equation images. Due to the unavailability of mathematical
equation image data sets with their textual descriptions, we generate two data
sets for experimental purpose. To validate the effectiveness of our MED model,
we conduct a real-world experiment to see whether the students are able to
write equations by only reading or listening their textual descriptions or not.
Experiments conclude that the students are able to write most of the equations
correctly by reading their textual descriptions only.

Computer Vision and Pattern Recognition