Image-to-LaTeX Converter for Mathematical Formulas and Text
<|reference_start|>Image-to-LaTeX Converter for Mathematical Formulas and Text: In this project, we train a vision encoder-decoder model to generate LaTeX code from images of mathematical formulas and text. Utilizing a diverse collection of image-to-LaTeX data, we build two models: a base model with a Swin Transformer encoder and a GPT-2 decoder, trained on machine-generated images, and a fine-tuned version enhanced with Low-Rank Adaptation (LoRA) trained on handwritten formulas. We then compare the BLEU performance of our specialized model on a handwritten test set with other similar models, such as Pix2Text, TexTeller, and Sumen. Through this project, we contribute open-source models for converting images to LaTeX and provide from-scratch code for building these models with distributed training and GPU optimizations.<|reference_end|>
@article{gurgurov2024image-to-latex,
    title={Image-to-LaTeX Converter for Mathematical Formulas and Text},
    author={Daniil Gurgurov, Aleksey Morshnev},
    journal={arXiv preprint arXiv:2408.04015},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.04015},
    primaryClass={cs.CL cs.CV}
}