title: VisualWebInstruct: Scaling up Multimodal Instruction Data through Web
    Search
summary: Vision-Language Models have made significant progress on many
  perception-focused tasks, however, their progress on reasoning-focused tasks
  seem to be limited due to the lack of high-quality and diverse training data.
  In this work, we aim to address the scarcity issue of reasoning-focused
  multimodal datasets. We propose VisualWebInstruct - a novel approach that
  leverages search engine to create a diverse, and high-quality dataset spanning
  multiple disciplines like math, physics, finance, chemistry, etc. Starting with
  meticulously selected 30,000 seed images, we employ Google Image search to
  identify websites containing similar images. We collect and process the HTMLs
  from over 700K unique URL sources. Through a pipeline of content extraction,
  filtering and synthesis, we build a dataset of approximately 900K
  question-answer pairs, with 40% being visual QA pairs and the rest as text QA
  pairs. Models fine-tuned on VisualWebInstruct demonstrate significant
  performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point
  gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain.
  Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B
  parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath
  (55.7%). These remarkable results highlight the effectiveness of our dataset in
  enhancing VLMs' reasoning capabilities for complex multimodal tasks.
paper_authors._id: 67d387ff45b17e31c16d05d1
paper_authors.name: Yiming Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:49:09.988Z
paper_authors._id: 6721451d41cc176331607843
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HQ6_0b2Y-X1ORS6tAz2cq.png
paper_authors.fullname: Yiming Jia
paper_authors.type: user
paper_authors.user: jymmmmm
paper_authors._id: 67d387ff45b17e31c16d05d2
paper_authors.name: Jiachen Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-14T10:49:35.163Z
paper_authors._id: 640d7ec5fdeaae13907fc488
paper_authors.avatarUrl: /avatars/acf43ea155105a51c8612dacc4725091.svg
paper_authors.fullname: Jiachen Li
paper_authors.type: user
paper_authors.user: jiachenli-ucsb
paper_authors._id: 67d387ff45b17e31c16d05d3
paper_authors.name: Xiang Yue
paper_authors._id: 67d387ff45b17e31c16d05d4
paper_authors.name: Bo Li
paper_authors._id: 67d387ff45b17e31c16d05d5
paper_authors.name: Ping Nie
paper_authors._id: 67d387ff45b17e31c16d05d6
paper_authors.name: Kai Zou
paper_authors._id: 67d387ff45b17e31c16d05d7
paper_authors.name: Wenhu Chen
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-14T01:36:13.720Z
paper_authors._id: 6313a86154e6e5d9f0f94e04
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
paper_authors.fullname: Wenhu Chen
paper_authors.type: user
paper_authors.user: wenhu
paper_title: VisualWebInstruct: Scaling up Multimodal Instruction Data through Web
    Search
paper_summary: Vision-Language Models have made significant progress on many
  perception-focused tasks, however, their progress on reasoning-focused tasks
  seem to be limited due to the lack of high-quality and diverse training data.
  In this work, we aim to address the scarcity issue of reasoning-focused
  multimodal datasets. We propose VisualWebInstruct - a novel approach that
  leverages search engine to create a diverse, and high-quality dataset spanning
  multiple disciplines like math, physics, finance, chemistry, etc. Starting with
  meticulously selected 30,000 seed images, we employ Google Image search to
  identify websites containing similar images. We collect and process the HTMLs
  from over 700K unique URL sources. Through a pipeline of content extraction,
  filtering and synthesis, we build a dataset of approximately 900K
  question-answer pairs, with 40% being visual QA pairs and the rest as text QA
  pairs. Models fine-tuned on VisualWebInstruct demonstrate significant
  performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point
  gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain.
  Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B
  parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath
  (55.7%). These remarkable results highlight the effectiveness of our dataset in
  enhancing VLMs' reasoning capabilities for complex multimodal tasks.
paper_ai_keywords: Vision-Language Models VisualWebInstruct search engine question-answer pairs visual QA pairs text QA pairs fine-tuned Llava-OV-mid MAmmoTH-VL MAmmoTH-VL2 MMMU-Pro-std MathVerse DynaMath