title: ViGoR: Improving Visual Grounding of Large Vision Language Models with
    Fine-Grained Reward Modeling
summary: By combining natural language understanding and the generation capabilities
  and breadth of knowledge of large language models with image perception, recent
  large vision language models (LVLMs) have shown unprecedented reasoning
  capabilities in the real world. However, the generated text often suffers from
  inaccurate grounding in the visual input, resulting in errors such as
  hallucinating nonexistent scene elements, missing significant parts of the
  scene, and inferring incorrect attributes and relationships between objects. To
  address these issues, we introduce a novel framework, ViGoR (Visual Grounding
  Through Fine-Grained Reward Modeling) that utilizes fine-grained reward
  modeling to significantly enhance the visual grounding of LVLMs over
  pre-trained baselines. This improvement is efficiently achieved using much
  cheaper human evaluations instead of full supervisions, as well as automated
  methods. We show the effectiveness of our approach through numerous metrics on
  several benchmarks. Additionally, we construct a comprehensive and challenging
  dataset specifically designed to validate the visual grounding capabilities of
  LVLMs. Finally, we plan to release our human annotation comprising
  approximately 16,000 images and generated text pairs with fine-grained
  evaluations to contribute to related research in the community.
paper_authors._id: 65c986ee97971388a50d1989
paper_authors.name: Siming Yan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:13:54.736Z
paper_authors._id: 64a308c9d993d477fa66a787
paper_authors.avatarUrl: /avatars/d6757499174d5186f0e07b95f76a9608.svg
paper_authors.fullname: Siming yan
paper_authors.type: user
paper_authors.user: simingyan
paper_authors._id: 65c986ee97971388a50d198a
paper_authors.name: Min Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:14:09.514Z
paper_authors._id: 63fe48321193ba2c229b3b7b
paper_authors.avatarUrl: /avatars/d4974e5e9b6cc0f1345495cf3ecbd445.svg
paper_authors.fullname: Min Bai
paper_authors.type: user
paper_authors.user: min2209
paper_authors._id: 65c986ee97971388a50d198b
paper_authors.name: Weifeng Chen
paper_authors._id: 65c986ee97971388a50d198c
paper_authors.name: Xiong Zhou
paper_authors._id: 65c986ee97971388a50d198d
paper_authors.name: Qixing Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T07:02:16.333Z
paper_authors._id: 67f18d051358145ee5bb3a11
paper_authors.avatarUrl: /avatars/e5a8c12295d7609625c2b891d47c3112.svg
paper_authors.fullname: qixing huang
paper_authors.type: user
paper_authors.user: qixinghuang1
paper_authors._id: 65c986ee97971388a50d198e
paper_authors.name: Li Erran Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-12T16:15:10.871Z
paper_authors._id: 65c88641810ded95b4b49966
paper_authors.avatarUrl: /avatars/23ce791d67f6b715e8023fd4fe761481.svg
paper_authors.fullname: Li Erran Li
paper_authors.type: user
paper_authors.user: erranli
paper_title: ViGoR: Improving Visual Grounding of Large Vision Language Models with
    Fine-Grained Reward Modeling
paper_summary: By combining natural language understanding and the generation capabilities
  and breadth of knowledge of large language models with image perception, recent
  large vision language models (LVLMs) have shown unprecedented reasoning
  capabilities in the real world. However, the generated text often suffers from
  inaccurate grounding in the visual input, resulting in errors such as
  hallucinating nonexistent scene elements, missing significant parts of the
  scene, and inferring incorrect attributes and relationships between objects. To
  address these issues, we introduce a novel framework, ViGoR (Visual Grounding
  Through Fine-Grained Reward Modeling) that utilizes fine-grained reward
  modeling to significantly enhance the visual grounding of LVLMs over
  pre-trained baselines. This improvement is efficiently achieved using much
  cheaper human evaluations instead of full supervisions, as well as automated
  methods. We show the effectiveness of our approach through numerous metrics on
  several benchmarks. Additionally, we construct a comprehensive and challenging
  dataset specifically designed to validate the visual grounding capabilities of
  LVLMs. Finally, we plan to release our human annotation comprising
  approximately 16,000 images and generated text pairs with fine-grained
  evaluations to contribute to related research in the community.
paper_ai_summary: ViGoR improves visual grounding in large vision language models using fine-grained reward modeling, enhancing accuracy and reducing the need for full supervision.
paper_ai_keywords: large vision language models LVLMS visual grounding fine-grained reward modeling human evaluations automated methods benchmarks dataset