title: PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal
    Documents
summary: Recent advancements in Large Multimodal Models (LMMs) have leveraged
  extensive multimodal datasets to enhance capabilities in complex
  knowledge-driven tasks. However, persistent challenges in perceptual and
  reasoning errors limit their efficacy, particularly in interpreting intricate
  visual data and deducing multimodal relationships. Addressing these issues, we
  introduce a novel dataset format, PIN (Paired and INterleaved multimodal
  documents), designed to significantly improve both the depth and breadth of
  multimodal training. The PIN format is built on three foundational principles:
  knowledge intensity, scalability, and support for diverse training modalities.
  This innovative format combines markdown files and comprehensive images to
  enrich training data with a dense knowledge structure and versatile training
  strategies. We present PIN-14M, an open-source dataset comprising 14 million
  samples derived from a diverse range of Chinese and English sources, tailored
  to include complex web and scientific content. This dataset is constructed
  meticulously to ensure data quality and ethical integrity, aiming to facilitate
  advanced training strategies and improve model robustness against common
  multimodal training pitfalls. Our initial results, forming the basis of this
  technical report, suggest significant potential for the PIN format in refining
  LMM performance, with plans for future expansions and detailed evaluations of
  its impact on model capabilities.
paper_authors._id: 6674e8e6dbe75ba786077cc6
paper_authors.name: Junjie Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:05:45.156Z
paper_authors._id: 62579c55b98dcaa7e0de285d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62579c55b98dcaa7e0de285d/0YUd5nloul_bW9yolDGGo.jpeg
paper_authors.fullname: wangjunjie
paper_authors.type: user
paper_authors.user: wanng
paper_authors._id: 6674e8e6dbe75ba786077cc7
paper_authors.name: Yin Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T08:12:03.713Z
paper_authors._id: 62be79222dd4214c062ec09d
paper_authors.avatarUrl: /avatars/09f78f8fd533e13ab7e54b5468e22da4.svg
paper_authors.fullname: yin zhang
paper_authors.type: user
paper_authors.user: miracleyin
paper_authors._id: 6674e8e6dbe75ba786077cc8
paper_authors.name: Yatai Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T08:12:10.313Z
paper_authors._id: 64c8b0a2f3d2a59a431dbb8e
paper_authors.avatarUrl: /avatars/bf130fdca7a2a0fe394558e2bf22c920.svg
paper_authors.fullname: yatai ji
paper_authors.type: user
paper_authors.user: jiyatai
paper_authors._id: 6674e8e6dbe75ba786077cc9
paper_authors.name: Yuxiang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-22T07:12:27.425Z
paper_authors._id: 6239386535384c2bcccd2a4f
paper_authors.avatarUrl: /avatars/c62cefc6235210ca12b46beb62751d8b.svg
paper_authors.fullname: Yuxiang Zhang
paper_authors.type: user
paper_authors.user: Joelzhang
paper_authors._id: 6674e8e6dbe75ba786077cca
paper_authors.name: Chunyang Jiang
paper_authors._id: 6674e8e6dbe75ba786077ccb
paper_authors.name: Yubo Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-30T08:41:38.245Z
paper_authors._id: 636a35eff8d9af4aea181608
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/636a35eff8d9af4aea181608/s9GFJYd_QXVbg0Lb4JpKj.jpeg
paper_authors.fullname: yubo
paper_authors.type: user
paper_authors.user: ubowang
paper_authors._id: 6674e8e6dbe75ba786077ccc
paper_authors.name: Kang Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-18T08:55:17.312Z
paper_authors._id: 6578265ddea7e2122d02f6ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6578265ddea7e2122d02f6ba/Bh6JjoVF5ceLSjV7Z7nTk.jpeg
paper_authors.fullname: kang zhu
paper_authors.type: user
paper_authors.user: kangz
paper_authors._id: 6674e8e6dbe75ba786077ccd
paper_authors.name: Zekun Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-07T07:31:22.430Z
paper_authors._id: 6149a9e95347647e6bb68882
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6149a9e95347647e6bb68882/Jddln1FxScCeVgTSCNBpr.png
paper_authors.fullname: Zekun Moore Wang
paper_authors.type: user
paper_authors.user: ZenMoore
paper_authors._id: 6674e8e6dbe75ba786077cce
paper_authors.name: Tiezhen Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-21T08:24:20.397Z
paper_authors._id: 62d22496c58f969c152bcefd
paper_authors.avatarUrl: /avatars/76c3b70e312f25e1e610473475553c5c.svg
paper_authors.fullname: Tiezhen WANG
paper_authors.type: user
paper_authors.user: xianbao
paper_authors._id: 6674e8e6dbe75ba786077ccf
paper_authors.name: Wenhao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-13T20:25:50.587Z
paper_authors._id: 65bb11cb00a03997849e9e85
paper_authors.avatarUrl: /avatars/17022b0254192a837f4fe00d84389cda.svg
paper_authors.fullname: Wenhao Huang
paper_authors.type: user
paper_authors.user: StephenHuang
paper_authors._id: 6674e8e6dbe75ba786077cd0
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-24T07:33:11.409Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 6674e8e6dbe75ba786077cd1
paper_authors.name: Bei Chen
paper_authors._id: 6674e8e6dbe75ba786077cd2
paper_authors.name: Qunshu Lin
paper_authors._id: 6674e8e6dbe75ba786077cd3
paper_authors.name: Minghao Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T11:47:38.165Z
paper_authors._id: 6417d9ea8f689506e7148417
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6417d9ea8f689506e7148417/bAYcruWNw4WvmuQcGgcwC.jpeg
paper_authors.fullname: minghao
paper_authors.type: user
paper_authors.user: Liam-Liu
paper_authors._id: 6674e8e6dbe75ba786077cd4
paper_authors.name: Ge Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:05:47.149Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 6674e8e6dbe75ba786077cd5
paper_authors.name: Wenhu Chen
paper_title: PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal
    Documents
paper_summary: Recent advancements in Large Multimodal Models (LMMs) have leveraged
  extensive multimodal datasets to enhance capabilities in complex
  knowledge-driven tasks. However, persistent challenges in perceptual and
  reasoning errors limit their efficacy, particularly in interpreting intricate
  visual data and deducing multimodal relationships. Addressing these issues, we
  introduce a novel dataset format, PIN (Paired and INterleaved multimodal
  documents), designed to significantly improve both the depth and breadth of
  multimodal training. The PIN format is built on three foundational principles:
  knowledge intensity, scalability, and support for diverse training modalities.
  This innovative format combines markdown files and comprehensive images to
  enrich training data with a dense knowledge structure and versatile training
  strategies. We present PIN-14M, an open-source dataset comprising 14 million
  samples derived from a diverse range of Chinese and English sources, tailored
  to include complex web and scientific content. This dataset is constructed
  meticulously to ensure data quality and ethical integrity, aiming to facilitate
  advanced training strategies and improve model robustness against common
  multimodal training pitfalls. Our initial results, forming the basis of this
  technical report, suggest significant potential for the PIN format in refining
  LMM performance, with plans for future expansions and detailed evaluations of
  its impact on model capabilities.
paper_ai_summary: A new dataset format called PIN is introduced to enhance LMMs by improving perceptual and reasoning capabilities through diverse, high-quality multimodal training data.
paper_ai_keywords: Large Multimodal Models LMMs perceptual errors reasoning errors multimodal datasets multimodal relationships PIN format knowledge intensity scalability training modalities markdown files comprehensive images dense knowledge structure versatile training strategies PIN-14M data quality ethical integrity model robustness common multimodal training pitfalls