FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts
<|reference_start|>FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts: Existing benchmarks for visual question answering lack in visual grounding and complexity, particularly in evaluating spatial reasoning skills. We introduce FlowVQA, a novel benchmark aimed at assessing the capabilities of visual question-answering multimodal language models in reasoning with flowcharts as visual contexts. FlowVQA comprises 2,272 carefully generated and human-verified flowchart images from three distinct content sources, along with 22,413 diverse question-answer pairs, to test a spectrum of reasoning tasks, including information localization, decision-making, and logical progression. We conduct a thorough baseline evaluation on a suite of both open-source and proprietary multimodal language models using various strategies, followed by an analysis of directional bias. The results underscore the benchmark's potential as a vital tool for advancing the field of multimodal modeling, providing a focused and challenging environment for enhancing model performance in visual and logical reasoning tasks.<|reference_end|>
@article{singh2024flowvqa:,
    title={FlowVQA: Mapping Multimodal Logic in Visual Question Answering with
  Flowcharts},
    author={Shubhankar Singh, Purvi Chaurasia, Yerram Varun, Pranshu Pandya,
  Vatsal Gupta, Vivek Gupta, Dan Roth},
    journal={arXiv preprint arXiv:2406.19237},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.19237},
    primaryClass={cs.CL cs.CV cs.IR cs.LG}
}