2.5 Years in Class: A Multimodal Textbook for Vision-Language
  Pretraining
Compared to image-text pair data, interleaved corpora enable Vision-Language
Models (VLMs) to understand the world more naturally like humans. However, such
existing datasets are crawled from webpage, facing challenges like low
knowledge density, loose image-text relations, and poor logical coherence
between images. On the other hand, the internet hosts vast instructional videos
(e.g., online geometry courses) that are widely used by humans to learn
foundational subjects, yet these valuable resources remain underexplored in VLM
training. In this paper, we introduce a high-quality multimodal
textbook corpus with richer foundational knowledge for VLM pretraining. It
collects over 2.5 years of instructional videos, totaling 22,000 class hours.
We first use an LLM-proposed taxonomy to systematically gather instructional
videos. Then we progressively extract and refine visual (keyframes), audio
(ASR), and textual knowledge (OCR) from the videos, and organize as an
image-text interleaved corpus based on temporal order. Compared to its
counterparts, our video-centric textbook offers more coherent context, richer
knowledge, and better image-text alignment. Experiments demonstrate its superb
pretraining performance, particularly in knowledge- and reasoning-intensive
tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook
exhibit outstanding interleaved context awareness, leveraging visual and
textual cues in their few-shot context for task solving~Our code are
available at \url{https://github.com/DAMO-NLP-SG/multimodal_textbook}.
67776143829d5102834151fb
Wenqi Zhang
claimed_verified
2025-01-07T08:42:43.304Z
6485bd278d14bcd5cdbb7c8d
/avatars/1427cf1a72b5db0cb263ad45885cf925.svg
Wenqi Zhang
user
zwq2018
67776143829d5102834151fc
Hang Zhang
67776143829d5102834151fd
Xin Li
claimed_verified
2025-01-05T23:02:41.348Z
63913b120cf6b11c487ca31d
/avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
Xin Li
user
lixin4ever
67776143829d5102834151fe
Jiashuo Sun
admin_assigned
2025-01-06T13:30:36.562Z
64527f701a57e1179c1c3693
/avatars/25b2632d7aa9ce26d5d4924ecb00c4f4.svg
Jiashuo Sun
user
gasolsun
67776143829d5102834151ff
Yongliang Shen
admin_assigned
2025-01-06T13:30:57.854Z
5e1058e9fcf41d740b69966d
/avatars/ce74839ba871f2b54313a670a233ba82.svg
Yongliang Shen
user
tricktreat
67776143829d510283415200
Weiming Lu
67776143829d510283415201
Deli Zhao
67776143829d510283415202
Yueting Zhuang
67776143829d510283415203
Lidong Bing
claimed_verified
2025-01-05T23:02:43.918Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
2.5 Years in Class: A Multimodal Textbook for Vision-Language
  Pretraining
Compared to image-text pair data, interleaved corpora enable Vision-Language
Models (VLMs) to understand the world more naturally like humans. However, such
existing datasets are crawled from webpage, facing challenges like low
knowledge density, loose image-text relations, and poor logical coherence
between images. On the other hand, the internet hosts vast instructional videos
(e.g., online geometry courses) that are widely used by humans to learn
foundational subjects, yet these valuable resources remain underexplored in VLM
training. In this paper, we introduce a high-quality multimodal
textbook corpus with richer foundational knowledge for VLM pretraining. It
collects over 2.5 years of instructional videos, totaling 22,000 class hours.
We first use an LLM-proposed taxonomy to systematically gather instructional
videos. Then we progressively extract and refine visual (keyframes), audio
(ASR), and textual knowledge (OCR) from the videos, and organize as an
image-text interleaved corpus based on temporal order. Compared to its
counterparts, our video-centric textbook offers more coherent context, richer
knowledge, and better image-text alignment. Experiments demonstrate its superb
pretraining performance, particularly in knowledge- and reasoning-intensive
tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook
exhibit outstanding interleaved context awareness, leveraging visual and
textual cues in their few-shot context for task solving~Our code are
available at \url{https://github.com/DAMO-NLP-SG/multimodal_textbook}.
A new high-quality instructional video-based corpus improves VLM pretraining through more coherent context and richer knowledge, enhancing performance in knowledge-intensive tasks.
Vision-Language Models VLMs multimodal textbook instructional videos LLM-proposed taxonomy keyframes ASR OCR image-text interleaved corpus ScienceQA MathVista