title: Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
summary: Vision Language Models (VLMs) demonstrate remarkable proficiency in
  addressing a wide array of visual questions, which requires strong perception
  and reasoning faculties. Assessing these two competencies independently is
  crucial for model refinement, despite the inherent difficulty due to the
  intertwined nature of seeing and reasoning in existing VLMs. To tackle this
  issue, we present Prism, an innovative framework designed to disentangle the
  perception and reasoning processes involved in visual question solving. Prism
  comprises two distinct stages: a perception stage that utilizes a VLM to
  extract and articulate visual information in textual form, and a reasoning
  stage that formulates responses based on the extracted visual information using
  a Large Language Model (LLM). This modular design enables the systematic
  comparison and assessment of both proprietary and open-source VLM for their
  perception and reasoning strengths. Our analytical framework provides several
  valuable insights, underscoring Prism's potential as a cost-effective solution
  for vision-language tasks. By combining a streamlined VLM focused on perception
  with a powerful LLM tailored for reasoning, Prism achieves superior results in
  general vision-language tasks while substantially cutting down on training and
  operational expenses. Quantitative evaluations show that Prism, when configured
  with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
  par with VLMs 10 times larger on the rigorous multimodal benchmark MMStar.
  The project is released at: https://github.com/SparksJoe/Prism.
paper_authors._id: 6675136b3ae9e9c659a0266e
paper_authors.name: Yuxuan Qiao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:05:37.653Z
paper_authors._id: 663cd4e3c1b4f7297c5dd958
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/663cd4e3c1b4f7297c5dd958/XZdsR1DTc9Njh8XnesTg1.jpeg
paper_authors.fullname: Yuxuan Qiao
paper_authors.type: user
paper_authors.user: Yuxuan-Qiao
paper_authors._id: 6675136b3ae9e9c659a0266f
paper_authors.name: Haodong Duan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:57:12.383Z
paper_authors._id: 63ee1379190ddd6214efd73a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676546883247-noauth.png
paper_authors.fullname: HAODONG DUAN
paper_authors.type: user
paper_authors.user: KennyUTC
paper_authors._id: 6675136b3ae9e9c659a02670
paper_authors.name: Xinyu Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:00:18.009Z
paper_authors._id: 64f5f8dd9b17cd59c453c57f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f5f8dd9b17cd59c453c57f/MulhwLcePFUWUQel8LQZ8.jpeg
paper_authors.fullname: Xinyu Fang
paper_authors.type: user
paper_authors.user: nebulae09
paper_authors._id: 6675136b3ae9e9c659a02671
paper_authors.name: Junming Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T13:33:23.625Z
paper_authors._id: 653f1d243bd61358055ad51d
paper_authors.avatarUrl: /avatars/698c03b9a4bb69659d2ed594626e3895.svg
paper_authors.fullname: junmingyang
paper_authors.type: user
paper_authors.user: jmyang
paper_authors._id: 6675136b3ae9e9c659a02672
paper_authors.name: Lin Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:05:31.876Z
paper_authors._id: 64b02ec0e5000ae8a572ced5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b02ec0e5000ae8a572ced5/6ifLntBU2ICQK7SW8WxKU.png
paper_authors.fullname: Lin Chen
paper_authors.type: user
paper_authors.user: Lin-Chen
paper_authors._id: 6675136b3ae9e9c659a02673
paper_authors.name: Songyang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:57:59.726Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 6675136b3ae9e9c659a02674
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T07:58:34.023Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_authors._id: 6675136b3ae9e9c659a02675
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T10:19:04.717Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 6675136b3ae9e9c659a02676
paper_authors.name: Kai Chen
paper_title: Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs
paper_summary: Vision Language Models (VLMs) demonstrate remarkable proficiency in
  addressing a wide array of visual questions, which requires strong perception
  and reasoning faculties. Assessing these two competencies independently is
  crucial for model refinement, despite the inherent difficulty due to the
  intertwined nature of seeing and reasoning in existing VLMs. To tackle this
  issue, we present Prism, an innovative framework designed to disentangle the
  perception and reasoning processes involved in visual question solving. Prism
  comprises two distinct stages: a perception stage that utilizes a VLM to
  extract and articulate visual information in textual form, and a reasoning
  stage that formulates responses based on the extracted visual information using
  a Large Language Model (LLM). This modular design enables the systematic
  comparison and assessment of both proprietary and open-source VLM for their
  perception and reasoning strengths. Our analytical framework provides several
  valuable insights, underscoring Prism's potential as a cost-effective solution
  for vision-language tasks. By combining a streamlined VLM focused on perception
  with a powerful LLM tailored for reasoning, Prism achieves superior results in
  general vision-language tasks while substantially cutting down on training and
  operational expenses. Quantitative evaluations show that Prism, when configured
  with a vanilla 2B LLaVA and freely accessible GPT-3.5, delivers performance on
  par with VLMs 10 times larger on the rigorous multimodal benchmark MMStar.
  The project is released at: https://github.com/SparksJoe/Prism.
paper_ai_summary: Prism is a framework that separates perception and reasoning in Vision Language Models, achieving competitive performance with smaller models and reduced costs.
paper_ai_keywords: Vision Language Models Prism Large Language Model multimodal benchmark MMStar LLaVA GPT-3.5