title: 2.5 Years in Class: A Multimodal Textbook for Vision-Language
    Pretraining
summary: Compared to image-text pair data, interleaved corpora enable Vision-Language
  Models (VLMs) to understand the world more naturally like humans. However, such
  existing datasets are crawled from webpage, facing challenges like low
  knowledge density, loose image-text relations, and poor logical coherence
  between images. On the other hand, the internet hosts vast instructional videos
  (e.g., online geometry courses) that are widely used by humans to learn
  foundational subjects, yet these valuable resources remain underexplored in VLM
  training. In this paper, we introduce a high-quality multimodal
  textbook corpus with richer foundational knowledge for VLM pretraining. It
  collects over 2.5 years of instructional videos, totaling 22,000 class hours.
  We first use an LLM-proposed taxonomy to systematically gather instructional
  videos. Then we progressively extract and refine visual (keyframes), audio
  (ASR), and textual knowledge (OCR) from the videos, and organize as an
  image-text interleaved corpus based on temporal order. Compared to its
  counterparts, our video-centric textbook offers more coherent context, richer
  knowledge, and better image-text alignment. Experiments demonstrate its superb
  pretraining performance, particularly in knowledge- and reasoning-intensive
  tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook
  exhibit outstanding interleaved context awareness, leveraging visual and
  textual cues in their few-shot context for task solving~Our code are
  available at \url{https://github.com/DAMO-NLP-SG/multimodal_textbook}.
paper_authors._id: 67776143829d5102834151fb
paper_authors.name: Wenqi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-07T08:42:43.304Z
paper_authors._id: 6485bd278d14bcd5cdbb7c8d
paper_authors.avatarUrl: /avatars/1427cf1a72b5db0cb263ad45885cf925.svg
paper_authors.fullname: Wenqi Zhang
paper_authors.type: user
paper_authors.user: zwq2018
paper_authors._id: 67776143829d5102834151fc
paper_authors.name: Hang Zhang
paper_authors._id: 67776143829d5102834151fd
paper_authors.name: Xin Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:02:41.348Z
paper_authors._id: 63913b120cf6b11c487ca31d
paper_authors.avatarUrl: /avatars/aec44edd5470dd6e767e0a25efd6fb5d.svg
paper_authors.fullname: Xin Li
paper_authors.type: user
paper_authors.user: lixin4ever
paper_authors._id: 67776143829d5102834151fe
paper_authors.name: Jiashuo Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:30:36.562Z
paper_authors._id: 64527f701a57e1179c1c3693
paper_authors.avatarUrl: /avatars/25b2632d7aa9ce26d5d4924ecb00c4f4.svg
paper_authors.fullname: Jiashuo Sun
paper_authors.type: user
paper_authors.user: gasolsun
paper_authors._id: 67776143829d5102834151ff
paper_authors.name: Yongliang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T13:30:57.854Z
paper_authors._id: 5e1058e9fcf41d740b69966d
paper_authors.avatarUrl: /avatars/ce74839ba871f2b54313a670a233ba82.svg
paper_authors.fullname: Yongliang Shen
paper_authors.type: user
paper_authors.user: tricktreat
paper_authors._id: 67776143829d510283415200
paper_authors.name: Weiming Lu
paper_authors._id: 67776143829d510283415201
paper_authors.name: Deli Zhao
paper_authors._id: 67776143829d510283415202
paper_authors.name: Yueting Zhuang
paper_authors._id: 67776143829d510283415203
paper_authors.name: Lidong Bing
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-05T23:02:43.918Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_title: 2.5 Years in Class: A Multimodal Textbook for Vision-Language
    Pretraining
paper_summary: Compared to image-text pair data, interleaved corpora enable Vision-Language
  Models (VLMs) to understand the world more naturally like humans. However, such
  existing datasets are crawled from webpage, facing challenges like low
  knowledge density, loose image-text relations, and poor logical coherence
  between images. On the other hand, the internet hosts vast instructional videos
  (e.g., online geometry courses) that are widely used by humans to learn
  foundational subjects, yet these valuable resources remain underexplored in VLM
  training. In this paper, we introduce a high-quality multimodal
  textbook corpus with richer foundational knowledge for VLM pretraining. It
  collects over 2.5 years of instructional videos, totaling 22,000 class hours.
  We first use an LLM-proposed taxonomy to systematically gather instructional
  videos. Then we progressively extract and refine visual (keyframes), audio
  (ASR), and textual knowledge (OCR) from the videos, and organize as an
  image-text interleaved corpus based on temporal order. Compared to its
  counterparts, our video-centric textbook offers more coherent context, richer
  knowledge, and better image-text alignment. Experiments demonstrate its superb
  pretraining performance, particularly in knowledge- and reasoning-intensive
  tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook
  exhibit outstanding interleaved context awareness, leveraging visual and
  textual cues in their few-shot context for task solving~Our code are
  available at \url{https://github.com/DAMO-NLP-SG/multimodal_textbook}.
paper_ai_keywords: Vision-Language Models (VLMs) multimodal textbook instructional videos keyframes ASR (Automatic Speech Recognition) OCR (Optical Character Recognition) image-text interleaved corpus temporal order knowledge- and reasoning-intensive tasks ScienceQA MathVista interleaved context awareness