title: DreamLLM: Synergistic Multimodal Comprehension and Creation
summary: This paper presents DreamLLM, a learning framework that first achieves
  versatile Multimodal Large Language Models (MLLMs) empowered with frequently
  overlooked synergy between multimodal comprehension and creation. DreamLLM
  operates on two fundamental principles. The first focuses on the generative
  modeling of both language and image posteriors by direct sampling in the raw
  multimodal space. This approach circumvents the limitations and information
  loss inherent to external feature extractors like CLIP, and a more thorough
  multimodal understanding is obtained. Second, DreamLLM fosters the generation
  of raw, interleaved documents, modeling both text and image contents, along
  with unstructured layouts. This allows DreamLLM to learn all conditional,
  marginal, and joint multimodal distributions effectively. As a result, DreamLLM
  is the first MLLM capable of generating free-form interleaved content.
  Comprehensive experiments highlight DreamLLM's superior performance as a
  zero-shot multimodal generalist, reaping from the enhanced learning synergy.
paper_authors._id: 650bc2b324da30e773ea9762
paper_authors.name: Runpei Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T07:46:39.028Z
paper_authors._id: 6201fc5d91d53938a6432fbf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6201fc5d91d53938a6432fbf/VLs8ZYaZrop4KBpZn53fH.jpeg
paper_authors.fullname: Runpei Dong
paper_authors.type: user
paper_authors.user: RunpeiDong
paper_authors._id: 650bc2b324da30e773ea9763
paper_authors.name: Chunrui Han
paper_authors._id: 650bc2b324da30e773ea9764
paper_authors.name: Yuang Peng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-21T07:22:38.366Z
paper_authors._id: 631ee086c1a8269da39265c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/631ee086c1a8269da39265c6/wUa1epGtTGcUv2mvrLUcD.png
paper_authors.fullname: Yuang Peng
paper_authors.type: user
paper_authors.user: yuangpeng
paper_authors._id: 650bc2b324da30e773ea9765
paper_authors.name: Zekun Qi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T07:47:48.699Z
paper_authors._id: 63c3e8abc7d7f4c63a515a02
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c3e8abc7d7f4c63a515a02/npMHnVP2hHLbvoUGe7C4O.jpeg
paper_authors.fullname: Zekun Qi
paper_authors.type: user
paper_authors.user: qizekun
paper_authors._id: 650bc2b324da30e773ea9766
paper_authors.name: Zheng Ge
paper_authors._id: 650bc2b324da30e773ea9767
paper_authors.name: Jinrong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-21T09:26:48.880Z
paper_authors._id: 650bd10c1f2949b99e65ef23
paper_authors.avatarUrl: /avatars/8f36f844690b64bcc7c358063f2bdc3b.svg
paper_authors.fullname: Jinrong Yang
paper_authors.type: user
paper_authors.user: YancieYJR
paper_authors._id: 650bc2b324da30e773ea9768
paper_authors.name: Liang Zhao
paper_authors._id: 650bc2b324da30e773ea9769
paper_authors.name: Jianjian Sun
paper_authors._id: 650bc2b324da30e773ea976a
paper_authors.name: Hongyu Zhou
paper_authors._id: 650bc2b324da30e773ea976b
paper_authors.name: Haoran Wei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-24T08:09:54.188Z
paper_authors._id: 6436618aeef1f55654a9f458
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6436618aeef1f55654a9f458/OvxGtuDg2GAFG9As-2hzW.jpeg
paper_authors.fullname: Haoran Wei
paper_authors.type: user
paper_authors.user: HaoranWei
paper_authors._id: 650bc2b324da30e773ea976c
paper_authors.name: Xiangwen Kong
paper_authors._id: 650bc2b324da30e773ea976d
paper_authors.name: Xiangyu Zhang
paper_authors._id: 650bc2b324da30e773ea976e
paper_authors.name: Kaisheng Ma
paper_authors._id: 650bc2b324da30e773ea976f
paper_authors.name: Li Yi
paper_title: DreamLLM: Synergistic Multimodal Comprehension and Creation
paper_summary: This paper presents DreamLLM, a learning framework that first achieves
  versatile Multimodal Large Language Models (MLLMs) empowered with frequently
  overlooked synergy between multimodal comprehension and creation. DreamLLM
  operates on two fundamental principles. The first focuses on the generative
  modeling of both language and image posteriors by direct sampling in the raw
  multimodal space. This approach circumvents the limitations and information
  loss inherent to external feature extractors like CLIP, and a more thorough
  multimodal understanding is obtained. Second, DreamLLM fosters the generation
  of raw, interleaved documents, modeling both text and image contents, along
  with unstructured layouts. This allows DreamLLM to learn all conditional,
  marginal, and joint multimodal distributions effectively. As a result, DreamLLM
  is the first MLLM capable of generating free-form interleaved content.
  Comprehensive experiments highlight DreamLLM's superior performance as a
  zero-shot multimodal generalist, reaping from the enhanced learning synergy.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) generative modeling raw multimodal space CLIP multimodal comprehension multimodal creation direct sampling information loss external feature extraction interleaved documents unstructured layouts conditional multimodal modeling marginal multimodal modeling joint multimodal modeling free-form interleaved content zero-shot multimodal generalist learning synergy