title: VisOnlyQA: Large Vision Language Models Still Struggle with Visual
    Perception of Geometric Information
summary: Errors in understanding visual information in images (i.e., visual perception
  errors) remain a major source of mistakes in Large Vision Language Models
  (LVLMs). While further analysis is essential, there is a deficiency in datasets
  for evaluating the visual perception of LVLMs. In this work, we introduce
  VisOnlyQA, a new dataset designed to directly evaluate the visual perception
  capabilities of LVLMs on questions about geometric and numerical information in
  scientific figures. Our dataset enables us to analyze the visual perception of
  LVLMs for fine-grained visual information, independent of other capabilities
  such as reasoning. The evaluation set of VisOnlyQA includes 1,200
  multiple-choice questions in 12 tasks on four categories of figures. We also
  provide synthetic training data consisting of 70k instances. Our experiments on
  VisOnlyQA highlight the following findings: (i) 20 LVLMs we evaluate, including
  GPT-4o and Gemini 1.5 Pro, work poorly on the visual perception tasks in
  VisOnlyQA, while human performance is nearly perfect. (ii) Fine-tuning on
  synthetic training data demonstrates the potential for enhancing the visual
  perception of LVLMs, but observed improvements are limited to certain tasks and
  specific models. (iii) Stronger language models improve the visual perception
  of LVLMs. In summary, our experiments suggest that both training data and model
  architectures should be improved to enhance the visual perception capabilities
  of LVLMs. The datasets, code, and model responses are provided at
  https://github.com/psunlpgroup/VisOnlyQA.
paper_authors._id: 674e87d63efd09e5cf7de76e
paper_authors.name: Ryo Kamoi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-03T11:15:03.159Z
paper_authors._id: 60591896102f61b42f65ae37
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/60591896102f61b42f65ae37/PSHcLoEeNnwvzAPOd6-6i.jpeg
paper_authors.fullname: Ryo Kamoi
paper_authors.type: user
paper_authors.user: ryokamoi
paper_authors._id: 674e87d63efd09e5cf7de76f
paper_authors.name: Yusen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:13:40.934Z
paper_authors._id: 6226742d7551f131b4eba59b
paper_authors.avatarUrl: /avatars/252f1fb7a82d34df4855241b5c8145ce.svg
paper_authors.fullname: Yusen Zhang
paper_authors.type: user
paper_authors.user: chatc
paper_authors._id: 674e87d63efd09e5cf7de770
paper_authors.name: Sarkar Snigdha Sarathi Das
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-03T16:13:46.969Z
paper_authors._id: 63c707ec656e7822e23af13c
paper_authors.avatarUrl: /avatars/50d2cfe1e55fe8bd2e0f09e2ca5715ea.svg
paper_authors.fullname: Sarkar Snigdha Sarathi Das
paper_authors.type: user
paper_authors.user: sarathismg
paper_authors._id: 674e87d63efd09e5cf7de771
paper_authors.name: Ranran Haoran Zhang
paper_authors._id: 674e87d63efd09e5cf7de772
paper_authors.name: Rui Zhang
paper_title: VisOnlyQA: Large Vision Language Models Still Struggle with Visual
    Perception of Geometric Information
paper_summary: Errors in understanding visual information in images (i.e., visual perception
  errors) remain a major source of mistakes in Large Vision Language Models
  (LVLMs). While further analysis is essential, there is a deficiency in datasets
  for evaluating the visual perception of LVLMs. In this work, we introduce
  VisOnlyQA, a new dataset designed to directly evaluate the visual perception
  capabilities of LVLMs on questions about geometric and numerical information in
  scientific figures. Our dataset enables us to analyze the visual perception of
  LVLMs for fine-grained visual information, independent of other capabilities
  such as reasoning. The evaluation set of VisOnlyQA includes 1,200
  multiple-choice questions in 12 tasks on four categories of figures. We also
  provide synthetic training data consisting of 70k instances. Our experiments on
  VisOnlyQA highlight the following findings: (i) 20 LVLMs we evaluate, including
  GPT-4o and Gemini 1.5 Pro, work poorly on the visual perception tasks in
  VisOnlyQA, while human performance is nearly perfect. (ii) Fine-tuning on
  synthetic training data demonstrates the potential for enhancing the visual
  perception of LVLMs, but observed improvements are limited to certain tasks and
  specific models. (iii) Stronger language models improve the visual perception
  of LVLMs. In summary, our experiments suggest that both training data and model
  architectures should be improved to enhance the visual perception capabilities
  of LVLMs. The datasets, code, and model responses are provided at
  https://github.com/psunlpgroup/VisOnlyQA.
paper_ai_summary: VisOnlyQA is a novel dataset designed to evaluate the fine-grained visual perception capabilities of Large Vision Language Models (LVLMs) on scientific figures, revealing that current models perform poorly and suggesting improvements in training data and architecture are necessary.
paper_ai_keywords: Large Vision Language Models (LVLMs) VisOnlyQA multiple-choice questions synthetic training data visual perception fine-grained visual information GPT-4o Gemini 1.5 Pro language models