title: UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond
    Scaling
summary: Significant research efforts have been made to scale and improve
  vision-language model (VLM) training approaches. Yet, with an ever-growing
  number of benchmarks, researchers are tasked with the heavy burden of
  implementing each protocol, bearing a non-trivial computational cost, and
  making sense of how all these benchmarks translate into meaningful axes of
  progress. To facilitate a systematic evaluation of VLM progress, we introduce
  UniBench: a unified implementation of 50+ VLM benchmarks spanning a
  comprehensive range of carefully categorized capabilities from object
  recognition to spatial awareness, counting, and much more. We showcase the
  utility of UniBench for measuring progress by evaluating nearly 60 publicly
  available vision-language models, trained on scales of up to 12.8B samples. We
  find that while scaling training data or model size can boost many
  vision-language model capabilities, scaling offers little benefit for reasoning
  or relations. Surprisingly, we also discover today's best VLMs struggle on
  simple digit recognition and counting tasks, e.g. MNIST, which much simpler
  networks can solve. Where scale falls short, we find that more precise
  interventions, such as data quality or tailored-learning objectives offer more
  promise. For practitioners, we also offer guidance on selecting a suitable VLM
  for a given application. Finally, we release an easy-to-run UniBench code-base
  with the full set of 50+ benchmarks and comparisons across 59 models as well as
  a distilled, representative set of benchmarks that runs in 5 minutes on a
  single GPU.
paper_authors._id: 66b97d8af1dfeeafae99200f
paper_authors.name: Haider Al-Tahan
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-12T03:12:13.598Z
paper_authors._id: 63654c5edaa65e06f90cc5ee
paper_authors.avatarUrl: /avatars/11fd0c361531f97b16b91629d0083b92.svg
paper_authors.fullname: haider
paper_authors.type: user
paper_authors.user: haideraltahan
paper_authors._id: 66b97d8af1dfeeafae992010
paper_authors.name: Quentin Garrido
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:11:12.158Z
paper_authors._id: 63049022412a1b9d381b9dcb
paper_authors.avatarUrl: /avatars/cac2a540fda40c5ae558968c50385f46.svg
paper_authors.fullname: Quentin Garrido
paper_authors.type: user
paper_authors.user: garridoq
paper_authors._id: 66b97d8af1dfeeafae992011
paper_authors.name: Randall Balestriero
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:11:18.506Z
paper_authors._id: 6629cf0121e410e67c4de179
paper_authors.avatarUrl: /avatars/157637f7716849675fe44da005fd3ede.svg
paper_authors.fullname: Randall Balestriero
paper_authors.type: user
paper_authors.user: RandallBalestriero
paper_authors._id: 66b97d8af1dfeeafae992012
paper_authors.name: Diane Bouchacourt
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:11:24.143Z
paper_authors._id: 641e01d13bae5a776368311b
paper_authors.avatarUrl: /avatars/1e983fb59f475b3252e201f44eec4657.svg
paper_authors.fullname: Diane Bouchacourt
paper_authors.type: user
paper_authors.user: DianeB
paper_authors._id: 66b97d8af1dfeeafae992013
paper_authors.name: Caner Hazirbas
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T11:11:29.894Z
paper_authors._id: 6515b9414eb20107d01eba58
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6515b9414eb20107d01eba58/8LSqNIgItKUVkWlEQERMD.jpeg
paper_authors.fullname: Caner Hazirbas
paper_authors.type: user
paper_authors.user: hazirbas
paper_authors._id: 66b97d8af1dfeeafae992014
paper_authors.name: Mark Ibrahim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-18T15:20:04.874Z
paper_authors._id: 64c17345e82e55936cf971bc
paper_authors.avatarUrl: /avatars/b39c5637e00291410620afb7c770d587.svg
paper_authors.fullname: Mark Ibrahim
paper_authors.type: user
paper_authors.user: marksibrahim
paper_title: UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond
    Scaling
paper_summary: Significant research efforts have been made to scale and improve
  vision-language model (VLM) training approaches. Yet, with an ever-growing
  number of benchmarks, researchers are tasked with the heavy burden of
  implementing each protocol, bearing a non-trivial computational cost, and
  making sense of how all these benchmarks translate into meaningful axes of
  progress. To facilitate a systematic evaluation of VLM progress, we introduce
  UniBench: a unified implementation of 50+ VLM benchmarks spanning a
  comprehensive range of carefully categorized capabilities from object
  recognition to spatial awareness, counting, and much more. We showcase the
  utility of UniBench for measuring progress by evaluating nearly 60 publicly
  available vision-language models, trained on scales of up to 12.8B samples. We
  find that while scaling training data or model size can boost many
  vision-language model capabilities, scaling offers little benefit for reasoning
  or relations. Surprisingly, we also discover today's best VLMs struggle on
  simple digit recognition and counting tasks, e.g. MNIST, which much simpler
  networks can solve. Where scale falls short, we find that more precise
  interventions, such as data quality or tailored-learning objectives offer more
  promise. For practitioners, we also offer guidance on selecting a suitable VLM
  for a given application. Finally, we release an easy-to-run UniBench code-base
  with the full set of 50+ benchmarks and comparisons across 59 models as well as
  a distilled, representative set of benchmarks that runs in 5 minutes on a
  single GPU.
paper_ai_summary: UniBench provides a unified evaluation framework for vision-language models across various benchmarks, revealing that scaling does not always improve reasoning capabilities and simpler networks can outperform large models on basic tasks.
paper_ai_keywords: vision-language model (VLM) UniBench benchmarks object recognition spatial awareness counting data quality learning objectives