MMR: Evaluating Reading Ability of Large Multimodal Models
<|reference_start|>MMR: Evaluating Reading Ability of Large Multimodal Models: Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Most existing text-rich image benchmarks are simple extraction-based question answering, and many LMMs now easily achieve high scores. This means that current benchmarks fail to accurately reflect performance of different models, and a natural idea is to build a new benchmark to evaluate their complex reasoning and spatial understanding abilities. In this work, we propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding. MMR is the first text-rich image benchmark built on human annotations with the help of language models. By evaluating several state-of-the-art LMMs, including GPT-4o, it reveals the limited capabilities of existing LMMs underscoring the value of our benchmark.<|reference_end|>
@article{chen2024mmr:,
    title={MMR: Evaluating Reading Ability of Large Multimodal Models},
    author={Jian Chen, Ruiyi Zhang, Yufan Zhou, Ryan Rossi, Jiuxiang Gu, Changyou
  Chen},
    journal={arXiv preprint arXiv:2408.14594},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.14594},
    primaryClass={cs.CV}
}