title: Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
summary: We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a
  vision-centric approach. While stronger language models can enhance multimodal
  capabilities, the design choices for vision components are often insufficiently
  explored and disconnected from visual representation learning research. This
  gap hinders accurate sensory grounding in real-world scenarios. Our study uses
  LLMs and visual instruction tuning as an interface to evaluate various visual
  representations, offering new insights into different models and architectures
  -- self-supervised, strongly supervised, or combinations thereof -- based on
  experiments with over 20 vision encoders. We critically examine existing MLLM
  benchmarks, addressing the difficulties involved in consolidating and
  interpreting results from various tasks, and introduce a new vision-centric
  benchmark, CV-Bench. To further improve visual grounding, we propose the
  Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that
  integrates high-resolution vision features with LLMs while reducing the number
  of tokens. Additionally, we discuss the curation of high-quality visual
  instruction-tuning data from publicly available sources, emphasizing the
  importance of data source balancing and distribution ratio. Collectively,
  Cambrian-1 not only achieves state-of-the-art performance but also serves as a
  comprehensive, open cookbook for instruction-tuned MLLMs. We provide model
  weights, code, supporting tools, datasets, and detailed instruction-tuning and
  evaluation recipes. We hope our release will inspire and accelerate
  advancements in multimodal systems and visual representation learning.
paper_authors._id: 667a74223463f1209e7ec98f
paper_authors.name: Shengbang Tong
paper_authors._id: 667a74223463f1209e7ec990
paper_authors.name: Ellis Brown
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-25T08:17:45.589Z
paper_authors._id: 626dc5105f7327906f0b2a4e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626dc5105f7327906f0b2a4e/QCSzuwYqsv8ozRnusVb-F.jpeg
paper_authors.fullname: Ellis Brown
paper_authors.type: user
paper_authors.user: ellisbrown
paper_authors._id: 667a74223463f1209e7ec991
paper_authors.name: Penghao Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:36:22.748Z
paper_authors._id: 64101f81b27543634e377fc1
paper_authors.avatarUrl: /avatars/557dd9d4707e3b38e0805dfb87c08004.svg
paper_authors.fullname: Penghao Wu
paper_authors.type: user
paper_authors.user: craigwu
paper_authors._id: 667a74223463f1209e7ec992
paper_authors.name: Sanghyun Woo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:36:28.657Z
paper_authors._id: 65da437eb62d242ed8f5ffcc
paper_authors.avatarUrl: /avatars/94483f3a1fd26ee43b32510538f1cd80.svg
paper_authors.fullname: Sanghyun Woo
paper_authors.type: user
paper_authors.user: shwoo93
paper_authors._id: 667a74223463f1209e7ec993
paper_authors.name: Manoj Middepogu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:36:33.894Z
paper_authors._id: 64284d6e6268f155b77ba3c5
paper_authors.avatarUrl: /avatars/4b3655264f8119f3a9f22bd547f0100b.svg
paper_authors.fullname: Manoj Middepogu
paper_authors.type: user
paper_authors.user: manoj1729
paper_authors._id: 667a74223463f1209e7ec994
paper_authors.name: Sai Charitha Akula
paper_authors._id: 667a74223463f1209e7ec995
paper_authors.name: Jihan Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:37:17.179Z
paper_authors._id: 6304baf041387c7f1177a5d2
paper_authors.avatarUrl: /avatars/795c63f2394080eec78ca7981d4a1f78.svg
paper_authors.fullname: Jihan Yang
paper_authors.type: user
paper_authors.user: jihanyang
paper_authors._id: 667a74223463f1209e7ec996
paper_authors.name: Shusheng Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:37:31.788Z
paper_authors._id: 627ccf058b4e56cfc2716425
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652346592327-noauth.jpeg
paper_authors.fullname: Shusheng Yang
paper_authors.type: user
paper_authors.user: ShushengYang
paper_authors._id: 667a74223463f1209e7ec997
paper_authors.name: Adithya Iyer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-26T08:05:01.569Z
paper_authors._id: 63fe5fb31193ba2c229c73bf
paper_authors.avatarUrl: /avatars/b5062ebc056f0659db12c6f6f463984b.svg
paper_authors.fullname: Adithya Iyer
paper_authors.type: user
paper_authors.user: Adithyaiyer1999
paper_authors._id: 667a74223463f1209e7ec998
paper_authors.name: Xichen Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:38:09.129Z
paper_authors._id: 63172831c92fd6fee3181f50
paper_authors.avatarUrl: /avatars/0f57068a138cb181e9451bfc1ed3d1c0.svg
paper_authors.fullname: Xichen Pan
paper_authors.type: user
paper_authors.user: xcpan
paper_authors._id: 667a74223463f1209e7ec999
paper_authors.name: Austin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:49:07.327Z
paper_authors._id: 64249f76d476e4ad55665d59
paper_authors.avatarUrl: /avatars/a0fec7e423ffae944a874ca267b55c1f.svg
paper_authors.fullname: Ziteng Wang
paper_authors.type: user
paper_authors.user: AustinWang0330
paper_authors._id: 667a74223463f1209e7ec99a
paper_authors.name: Rob Fergus
paper_authors._id: 667a74223463f1209e7ec99b
paper_authors.name: Yann LeCun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:39:07.946Z
paper_authors._id: 64ed0b8c2203a126eb1a5b9a
paper_authors.avatarUrl: /avatars/9156dc406ed3f9ee62b73657ac20f5ed.svg
paper_authors.fullname: Yann LeCun
paper_authors.type: user
paper_authors.user: ylecun
paper_authors._id: 667a74223463f1209e7ec99c
paper_authors.name: Saining Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-25T08:39:00.996Z
paper_authors._id: 6596422646624a86ff3b3bda
paper_authors.avatarUrl: /avatars/216e12b77e45ac5f1fa20932f5745411.svg
paper_authors.fullname: Saining Xie
paper_authors.type: user
paper_authors.user: sainx
paper_title: Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
paper_summary: We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a
  vision-centric approach. While stronger language models can enhance multimodal
  capabilities, the design choices for vision components are often insufficiently
  explored and disconnected from visual representation learning research. This
  gap hinders accurate sensory grounding in real-world scenarios. Our study uses
  LLMs and visual instruction tuning as an interface to evaluate various visual
  representations, offering new insights into different models and architectures
  -- self-supervised, strongly supervised, or combinations thereof -- based on
  experiments with over 20 vision encoders. We critically examine existing MLLM
  benchmarks, addressing the difficulties involved in consolidating and
  interpreting results from various tasks, and introduce a new vision-centric
  benchmark, CV-Bench. To further improve visual grounding, we propose the
  Spatial Vision Aggregator (SVA), a dynamic and spatially-aware connector that
  integrates high-resolution vision features with LLMs while reducing the number
  of tokens. Additionally, we discuss the curation of high-quality visual
  instruction-tuning data from publicly available sources, emphasizing the
  importance of data source balancing and distribution ratio. Collectively,
  Cambrian-1 not only achieves state-of-the-art performance but also serves as a
  comprehensive, open cookbook for instruction-tuned MLLMs. We provide model
  weights, code, supporting tools, datasets, and detailed instruction-tuning and
  evaluation recipes. We hope our release will inspire and accelerate
  advancements in multimodal systems and visual representation learning.
paper_ai_keywords: multimodal LLMs (MLLMs) vision-centric approach visual representation learning LLMs and visual instruction tuning visual instruction-tuning vision encoders CV-Bench Spatial Vision Aggregator (SVA) dynamic and spatially-aware connector high-resolution vision features instruction-tuned MLLMs