title: VILA^2: VILA Augmented VILA
summary: Visual language models (VLMs) have rapidly progressed, driven by the success
  of large language models (LLMs). While model architectures and training
  infrastructures advance rapidly, data curation remains under-explored. When
  data quantity and quality become a bottleneck, existing work either directly
  crawls more raw data from the Internet that does not have a guarantee of data
  quality or distills from black-box commercial models (e.g., GPT-4V / Gemini)
  causing the performance upper bounded by that model. In this work, we introduce
  a novel approach that includes a self-augment step and a specialist-augment
  step to iteratively improve data quality and model performance. In the
  self-augment step, a VLM recaptions its own pretraining data to enhance data
  quality, and then retrains from scratch using this refined dataset to improve
  model performance. This process can iterate for several rounds. Once
  self-augmentation saturates, we employ several specialist VLMs finetuned from
  the self-augmented VLM with domain-specific expertise, to further infuse
  specialist knowledge into the generalist VLM through task-oriented recaptioning
  and retraining. With the combined self-augmented and specialist-augmented
  training, we introduce VILA^2 (VILA-augmented-VILA), a VLM family that
  consistently improves the accuracy on a wide range of tasks over prior art, and
  achieves new state-of-the-art results on MMMU leaderboard among open-sourced
  models.
paper_authors._id: 66a1bbb7ca8ee359d60e5118
paper_authors.name: Yunhao Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-25T07:38:26.772Z
paper_authors._id: 640d68938aee167ccda391da
paper_authors.avatarUrl: /avatars/8e3fbf6ca10fe4e9b04a3a84d4e3e255.svg
paper_authors.fullname: Yunhao Fang
paper_authors.type: user
paper_authors.user: Seerkfang
paper_authors._id: 66a1bbb7ca8ee359d60e5119
paper_authors.name: Ligeng Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T08:19:26.721Z
paper_authors._id: 62b4b5beb25cb80fcf278354
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62b4b5beb25cb80fcf278354/7uGNOeE91paZikhK2QFgp.jpeg
paper_authors.fullname: Ligeng Zhu
paper_authors.type: user
paper_authors.user: Ligeng-Zhu
paper_authors._id: 66a1bbb7ca8ee359d60e511a
paper_authors.name: Yao Lu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T08:19:39.594Z
paper_authors._id: 64119ace518736207a51625a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64119ace518736207a51625a/bJ-hyscgoU7a9gs-br0H7.jpeg
paper_authors.fullname: Yao Lu
paper_authors.type: user
paper_authors.user: yaolu
paper_authors._id: 66a1bbb7ca8ee359d60e511b
paper_authors.name: Yan Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:40:37.179Z
paper_authors._id: 64b9c32c2ab555ea562c736c
paper_authors.avatarUrl: /avatars/b22f6d77aaeb1e6dd34aa3c4d53f1e3e.svg
paper_authors.fullname: Yan Wang
paper_authors.type: user
paper_authors.user: yyyanw
paper_authors._id: 66a1bbb7ca8ee359d60e511c
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:08:12.215Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_authors._id: 66a1bbb7ca8ee359d60e511d
paper_authors.name: Jang Hyun Cho
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T08:20:49.312Z
paper_authors._id: 647a4ae6c6077fb66fbd0036
paper_authors.avatarUrl: /avatars/ee08791366311da83bd5395012f3b1c2.svg
paper_authors.fullname: Jang Hyun Cho
paper_authors.type: user
paper_authors.user: janghyuncho7
paper_authors._id: 66a1bbb7ca8ee359d60e511e
paper_authors.name: Marco Pavone
paper_authors._id: 66a1bbb7ca8ee359d60e511f
paper_authors.name: Song Han
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T08:21:16.081Z
paper_authors._id: 63797f727df2fefdcaf3ff7e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668906853549-noauth.jpeg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: songhan
paper_authors._id: 66a1bbb7ca8ee359d60e5120
paper_authors.name: Hongxu Yin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-25T08:21:26.595Z
paper_authors._id: 65a8b7f69aec1645994e7a15
paper_authors.avatarUrl: /avatars/debc086f3fea029db22847bde80799a0.svg
paper_authors.fullname: Hongxu Yin
paper_authors.type: user
paper_authors.user: yinhongxu
paper_title: VILA^2: VILA Augmented VILA
paper_summary: Visual language models (VLMs) have rapidly progressed, driven by the success
  of large language models (LLMs). While model architectures and training
  infrastructures advance rapidly, data curation remains under-explored. When
  data quantity and quality become a bottleneck, existing work either directly
  crawls more raw data from the Internet that does not have a guarantee of data
  quality or distills from black-box commercial models (e.g., GPT-4V / Gemini)
  causing the performance upper bounded by that model. In this work, we introduce
  a novel approach that includes a self-augment step and a specialist-augment
  step to iteratively improve data quality and model performance. In the
  self-augment step, a VLM recaptions its own pretraining data to enhance data
  quality, and then retrains from scratch using this refined dataset to improve
  model performance. This process can iterate for several rounds. Once
  self-augmentation saturates, we employ several specialist VLMs finetuned from
  the self-augmented VLM with domain-specific expertise, to further infuse
  specialist knowledge into the generalist VLM through task-oriented recaptioning
  and retraining. With the combined self-augmented and specialist-augmented
  training, we introduce VILA^2 (VILA-augmented-VILA), a VLM family that
  consistently improves the accuracy on a wide range of tasks over prior art, and
  achieves new state-of-the-art results on MMMU leaderboard among open-sourced
  models.
paper_ai_keywords: Visual language models (VLMs) Large language models (LLMs) Data curation Data quality Raw data Black-box commercial models GPT-4V Gemini Self-augment step Specialist-augment step Pretraining data Recaptions Retraining Finetuned VLMs Task-oriented recaptioning $VILA^2$ (VILA-augmented-VILA) MMMU leaderboard State-of-the-art results