Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark
<|reference_start|>Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark: We evaluate the zero-shot ability of GPT-4 and LLaVa to perform simple Visual Network Analysis (VNA) tasks on small-scale graphs. We evaluate the Vision Language Models (VLMs) on 5 tasks related to three foundational network science concepts: identifying nodes of maximal degree on a rendered graph, identifying whether signed triads are balanced or unbalanced, and counting components. The tasks are structured to be easy for a human who understands the underlying graph theoretic concepts, and can all be solved by counting the appropriate elements in graphs. We find that while GPT-4 consistently outperforms LLaVa, both models struggle with every visual network analysis task we propose. We publicly release the first benchmark for the evaluation of VLMs on foundational VNA tasks.<|reference_end|>
@article{williams2024multimodal,
    title={Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA
  Benchmark},
    author={Evan M. Williams and Kathleen M. Carley},
    journal={arXiv preprint arXiv:2405.06634},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.06634},
    primaryClass={cs.CV cs.AI cs.CL}
}