title: VisIT-Bench: A Benchmark for Vision-Language Instruction Following
    Inspired by Real-World Use
summary: We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for
  evaluation of instruction-following vision-language models for real-world use.
  Our starting point is curating 70 'instruction families' that we envision
  instruction tuned vision-language models should be able to address. Extending
  beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to
  game playing and creative generation. Following curation, our dataset comprises
  592 test queries, each with a human-authored instruction-conditioned caption.
  These descriptions surface instruction-specific factors, e.g., for an
  instruction asking about the accessibility of a storefront for wheelchair
  users, the instruction-conditioned caption describes ramps/potential obstacles.
  These descriptions enable 1) collecting human-verified reference outputs for
  each instance; and 2) automatic evaluation of candidate multimodal generations
  using a text-only LLM, aligning with human judgment. We quantify quality gaps
  between models and references using both human and automatic evaluations; e.g.,
  the top-performing instruction-following model wins against the GPT-4 reference
  in just 27% of the comparison. VisIT-Bench is dynamic to participate,
  practitioners simply submit their model's response on the project website;
  Data, code and leaderboard is available at visit-bench.github.io.
paper_authors._id: 64db0bfc00b80a024c5924f7
paper_authors.name: Yonatan Bitton
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-15T12:21:14.755Z
paper_authors._id: 632e0771ae0a7b1fc95630bf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663961181981-632e0771ae0a7b1fc95630bf.jpeg
paper_authors.fullname: Yonatan
paper_authors.type: user
paper_authors.user: yonatanbitton
paper_authors._id: 64db0bfc00b80a024c5924f8
paper_authors.name: Hritik Bansal
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-15T12:56:40.216Z
paper_authors._id: 61c5c25705aa54027c52f7b3
paper_authors.avatarUrl: /avatars/8a89e040dc331b7a83d9a704c4fc29d2.svg
paper_authors.fullname: Hritik Bansal
paper_authors.type: user
paper_authors.user: hbXNov
paper_authors._id: 64db0bfc00b80a024c5924f9
paper_authors.name: Jack Hessel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:26:23.279Z
paper_authors._id: 625498644f4edf771516b2cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
paper_authors.fullname: Jack Hessel
paper_authors.type: user
paper_authors.user: jmhessel
paper_authors._id: 64db0bfc00b80a024c5924fa
paper_authors.name: Rulin Shao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:26:42.433Z
paper_authors._id: 6334a0bd31a2be3938c59537
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6334a0bd31a2be3938c59537/kSetFUWAmJbPQ1KSlNKBr.jpeg
paper_authors.fullname: Rulin Shao
paper_authors.type: user
paper_authors.user: rulins
paper_authors._id: 64db0bfc00b80a024c5924fb
paper_authors.name: Wanrong Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-16T13:23:22.828Z
paper_authors._id: 633a7e91c8eb7b089034fb3c
paper_authors.avatarUrl: /avatars/0e03d80aa469de131cc17bb0c5c016b9.svg
paper_authors.fullname: Wanrong Zhu
paper_authors.type: user
paper_authors.user: VegB
paper_authors._id: 64db0bfc00b80a024c5924fc
paper_authors.name: Anas Awadalla
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:27:09.283Z
paper_authors._id: 60f1ab3afe0d78e01037eeb1
paper_authors.avatarUrl: /avatars/f784fa423fd84fffb4683fa837ffc5a3.svg
paper_authors.fullname: Anas Awadalla
paper_authors.type: user
paper_authors.user: anas-awadalla
paper_authors._id: 64db0bfc00b80a024c5924fd
paper_authors.name: Josh Gardner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:27:47.014Z
paper_authors._id: 63e6d6b22d2c508de90153a1
paper_authors.avatarUrl: /avatars/11d653e5376bd122ff82778a8393f541.svg
paper_authors.fullname: Josh Gardner
paper_authors.type: user
paper_authors.user: jpgard
paper_authors._id: 64db0bfc00b80a024c5924fe
paper_authors.name: Rohan Taori
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-15T13:27:56.760Z
paper_authors._id: 622fca0d8d46dde5d605f83e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1647489114181-622fca0d8d46dde5d605f83e.jpeg
paper_authors.fullname: Rohan Taori
paper_authors.type: user
paper_authors.user: rtaori
paper_authors._id: 64db0bfc00b80a024c5924ff
paper_authors.name: Ludwig Schimdt
paper_title: VisIT-Bench: A Benchmark for Vision-Language Instruction Following
    Inspired by Real-World Use
paper_summary: We introduce VisIT-Bench (Visual InsTruction Benchmark), a benchmark for
  evaluation of instruction-following vision-language models for real-world use.
  Our starting point is curating 70 'instruction families' that we envision
  instruction tuned vision-language models should be able to address. Extending
  beyond evaluations like VQAv2 and COCO, tasks range from basic recognition to
  game playing and creative generation. Following curation, our dataset comprises
  592 test queries, each with a human-authored instruction-conditioned caption.
  These descriptions surface instruction-specific factors, e.g., for an
  instruction asking about the accessibility of a storefront for wheelchair
  users, the instruction-conditioned caption describes ramps/potential obstacles.
  These descriptions enable 1) collecting human-verified reference outputs for
  each instance; and 2) automatic evaluation of candidate multimodal generations
  using a text-only LLM, aligning with human judgment. We quantify quality gaps
  between models and references using both human and automatic evaluations; e.g.,
  the top-performing instruction-following model wins against the GPT-4 reference
  in just 27% of the comparison. VisIT-Bench is dynamic to participate,
  practitioners simply submit their model's response on the project website;
  Data, code and leaderboard is available at visit-bench.github.io.
paper_ai_summary: VisIT-Bench evaluates instruction-following vision-language models across a variety of tasks through human-authored instructions and multimodal generation assessments.
paper_ai_keywords: instruction-following vision-language models instruction families VQAv2 COCO instruction-conditioned caption multimodal generation automatic evaluation text-only LLM leaderboard