VisualWebInstruct: Scaling up Multimodal Instruction Data through Web
  Search
Vision-Language Models have made significant progress on many
perception-focused tasks, however, their progress on reasoning-focused tasks
seem to be limited due to the lack of high-quality and diverse training data.
In this work, we aim to address the scarcity issue of reasoning-focused
multimodal datasets. We propose VisualWebInstruct - a novel approach that
leverages search engine to create a diverse, and high-quality dataset spanning
multiple disciplines like math, physics, finance, chemistry, etc. Starting with
meticulously selected 30,000 seed images, we employ Google Image search to
identify websites containing similar images. We collect and process the HTMLs
from over 700K unique URL sources. Through a pipeline of content extraction,
filtering and synthesis, we build a dataset of approximately 900K
question-answer pairs, with 40% being visual QA pairs and the rest as text QA
pairs. Models fine-tuned on VisualWebInstruct demonstrate significant
performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point
gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain.
Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B
parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath
(55.7%). These remarkable results highlight the effectiveness of our dataset in
enhancing VLMs' reasoning capabilities for complex multimodal tasks.
67d387ff45b17e31c16d05d1
Yiming Jia
admin_assigned
2025-03-14T10:49:09.988Z
6721451d41cc176331607843
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/HQ6_0b2Y-X1ORS6tAz2cq.png
Yiming Jia
user
jymmmmm
67d387ff45b17e31c16d05d2
Jiachen Li
admin_assigned
2025-03-14T10:49:35.163Z
640d7ec5fdeaae13907fc488
/avatars/acf43ea155105a51c8612dacc4725091.svg
Jiachen Li
user
jiachenli-ucsb
67d387ff45b17e31c16d05d3
Xiang Yue
claimed_verified
2025-07-02T16:08:18.402Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
67d387ff45b17e31c16d05d4
Bo Li
67d387ff45b17e31c16d05d5
Ping Nie
67d387ff45b17e31c16d05d6
Kai Zou
67d387ff45b17e31c16d05d7
Wenhu Chen
extracted_pending
2025-03-14T01:36:13.720Z
6313a86154e6e5d9f0f94e04
https://cdn-avatars.huggingface.co/v1/production/uploads/1662232951344-6313a86154e6e5d9f0f94e04.jpeg
Wenhu Chen
user
wenhu
VisualWebInstruct: Scaling up Multimodal Instruction Data through Web
  Search
Vision-Language Models have made significant progress on many
perception-focused tasks, however, their progress on reasoning-focused tasks
seem to be limited due to the lack of high-quality and diverse training data.
In this work, we aim to address the scarcity issue of reasoning-focused
multimodal datasets. We propose VisualWebInstruct - a novel approach that
leverages search engine to create a diverse, and high-quality dataset spanning
multiple disciplines like math, physics, finance, chemistry, etc. Starting with
meticulously selected 30,000 seed images, we employ Google Image search to
identify websites containing similar images. We collect and process the HTMLs
from over 700K unique URL sources. Through a pipeline of content extraction,
filtering and synthesis, we build a dataset of approximately 900K
question-answer pairs, with 40% being visual QA pairs and the rest as text QA
pairs. Models fine-tuned on VisualWebInstruct demonstrate significant
performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point
gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain.
Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B
parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath
(55.7%). These remarkable results highlight the effectiveness of our dataset in
enhancing VLMs' reasoning capabilities for complex multimodal tasks.
The VisualWebInstruct approach enhances vision-language models' reasoning abilities through a large, diverse, and high-quality multimodal dataset created from search engine data.
vision-language models multimodal datasets question-answer pairs visual QA text QA Llava-OV-mid MAmmoTH-VL MAmmoTH-VL2 MMMU-Pro-std MathVerse DynaMath