A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models
<|reference_start|>A Cognitive Evaluation Benchmark of Image Reasoning and Description for Large Vision-Language Models: Large Vision-Language Models (LVLMs), despite their recent success, are hardly comprehensively tested for their cognitive abilities. Inspired by the prevalent use of the "Cookie Theft" task in human cognition test, we propose a novel evaluation benchmark to evaluate high-level cognitive ability of LVLMs using images with rich semantics. It defines eight reasoning capabilities and consists of an image description task and a visual question answering task. Our evaluation on well-known LVLMs shows that there is still a large gap in cognitive ability between LVLMs and humans.<|reference_end|>
@article{song2024a,
    title={A Cognitive Evaluation Benchmark of Image Reasoning and Description for
  Large Vision-Language Models},
    author={Xiujie Song, Mengyue Wu, Kenny Q. Zhu, Chunhao Zhang, Yanyi Chen},
    journal={arXiv preprint arXiv:2402.18409},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18409},
    primaryClass={cs.AI cs.CL cs.CV}
}