title: Prismatic VLMs: Investigating the Design Space of Visually-Conditioned
    Language Models
summary: Visually-conditioned language models (VLMs) have seen growing adoption in
  applications such as visual dialogue, scene understanding, and robotic task
  planning; adoption that has fueled a wealth of new models such as LLaVa,
  InstructBLIP, and PaLI-3. Despite the volume of new releases, key design
  decisions around image preprocessing, architecture, and optimization are
  under-explored, making it challenging to understand what factors account for
  model performance - a challenge further complicated by the lack of objective,
  consistent evaluations. To address these gaps, we first compile a suite of
  standardized evaluations spanning visual question answering, object
  localization from language, and targeted challenge sets that probe properties
  such as hallucination; evaluations that provide calibrated, fine-grained
  insight into a VLM's capabilities. Second, we rigorously investigate VLMs along
  key design axes, including pretrained visual representations and quantifying
  the tradeoffs of using base vs. instruct-tuned language models, amongst others.
  We couple our analysis with three resource contributions: (1) a unified
  framework for evaluating VLMs, (2) optimized, flexible code for VLM training,
  and (3) checkpoints for all models, including a family of VLMs at the 7-13B
  scale that strictly outperform InstructBLIP and LLaVa v1.5, the
  state-of-the-art in open-source VLMs.
paper_authors._id: 65cae4c8ab05dc46af7e5bef
paper_authors.name: Siddharth Karamcheti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:48:39.900Z
paper_authors._id: 6150b090d84cf0532aa1764b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634666324094-6150b090d84cf0532aa1764b.jpeg
paper_authors.fullname: Siddharth Karamcheti
paper_authors.type: user
paper_authors.user: skaramcheti
paper_authors._id: 65cae4c8ab05dc46af7e5bf0
paper_authors.name: Suraj Nair
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:49:24.667Z
paper_authors._id: 624f0e38c8b211c193d325dd
paper_authors.avatarUrl: /avatars/370b7b21bd1bc284d3f48c0c40dfa026.svg
paper_authors.fullname: Suraj Nair
paper_authors.type: user
paper_authors.user: surajnair
paper_authors._id: 65cae4c8ab05dc46af7e5bf1
paper_authors.name: Ashwin Balakrishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:49:43.603Z
paper_authors._id: 653ad464170f0a2a9202dbe3
paper_authors.avatarUrl: /avatars/dd48cf05b946663dcb927545cd8edddc.svg
paper_authors.fullname: Ashwin Balakrishna
paper_authors.type: user
paper_authors.user: abalakrishnaTRI
paper_authors._id: 65cae4c8ab05dc46af7e5bf2
paper_authors.name: Percy Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:49:50.742Z
paper_authors._id: 6409651b9e9f790c905b2335
paper_authors.avatarUrl: /avatars/1fb8c80b60f21f65a0a027319101f236.svg
paper_authors.fullname: Percy Liang
paper_authors.type: user
paper_authors.user: percyliang
paper_authors._id: 65cae4c8ab05dc46af7e5bf3
paper_authors.name: Thomas Kollar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T14:49:57.889Z
paper_authors._id: 64358e662d0ed796668cb438
paper_authors.avatarUrl: /avatars/6f86547d8691e061e289dd906deedfdb.svg
paper_authors.fullname: Thomas Kollar
paper_authors.type: user
paper_authors.user: thomas-kollar
paper_authors._id: 65cae4c8ab05dc46af7e5bf4
paper_authors.name: Dorsa Sadigh
paper_title: Prismatic VLMs: Investigating the Design Space of Visually-Conditioned
    Language Models
paper_summary: Visually-conditioned language models (VLMs) have seen growing adoption in
  applications such as visual dialogue, scene understanding, and robotic task
  planning; adoption that has fueled a wealth of new models such as LLaVa,
  InstructBLIP, and PaLI-3. Despite the volume of new releases, key design
  decisions around image preprocessing, architecture, and optimization are
  under-explored, making it challenging to understand what factors account for
  model performance - a challenge further complicated by the lack of objective,
  consistent evaluations. To address these gaps, we first compile a suite of
  standardized evaluations spanning visual question answering, object
  localization from language, and targeted challenge sets that probe properties
  such as hallucination; evaluations that provide calibrated, fine-grained
  insight into a VLM's capabilities. Second, we rigorously investigate VLMs along
  key design axes, including pretrained visual representations and quantifying
  the tradeoffs of using base vs. instruct-tuned language models, amongst others.
  We couple our analysis with three resource contributions: (1) a unified
  framework for evaluating VLMs, (2) optimized, flexible code for VLM training,
  and (3) checkpoints for all models, including a family of VLMs at the 7-13B
  scale that strictly outperform InstructBLIP and LLaVa v1.5, the
  state-of-the-art in open-source VLMs.
paper_ai_summary: The paper provides standardized evaluations and investigations into visually-conditioned language models (VLMs), exploring factors affecting their performance and offering resources like a unified evaluation framework and optimized training code.
paper_ai_keywords: visually-conditioned language models VLMs visual dialogue scene understanding robotic task planning LLaVa InstructBLIP PaLI-3 image preprocessing architecture optimization visual question answering object localization hallucination pretrained visual representations instruct-tuned language models unified framework VLM training checkpoints state-of-the-art