title: Multimodal Representation Alignment for Image Generation: Text-Image
    Interleaved Control Is Easier Than You Think
summary: The field of advanced text-to-image generation is witnessing the emergence of
  unified frameworks that integrate powerful text encoders, such as CLIP and T5,
  with Diffusion Transformer backbones. Although there have been efforts to
  control output images with additional conditions, like canny and depth map, a
  comprehensive framework for arbitrary text-image interleaved control is still
  lacking. This gap is especially evident when attempting to merge concepts or
  visual elements from multiple images in the generation process. To mitigate the
  gap, we conducted preliminary experiments showing that large multimodal models
  (LMMs) offer an effective shared representation space, where image and text can
  be well-aligned to serve as a condition for external diffusion models. Based on
  this discovery, we propose Dream Engine, an efficient and unified framework
  designed for arbitrary text-image interleaved control in image generation
  models. Building on powerful text-to-image models like SD3.5, we replace the
  original text-only encoders by incorporating versatile multimodal information
  encoders such as QwenVL. Our approach utilizes a two-stage training paradigm,
  consisting of joint text-image alignment and multimodal interleaved instruction
  tuning. Our experiments demonstrate that this training method is effective,
  achieving a 0.69 overall score on the GenEval benchmark, and matching the
  performance of state-of-the-art text-to-image models like SD3.5 and FLUX.
paper_authors._id: 67c17b8f60206395233b7e46
paper_authors.name: Liang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:34:40.397Z
paper_authors._id: 658c481dd1c8b106727a8b73
paper_authors.avatarUrl: /avatars/d34a7a62c3a524e5fdd2d5994348db58.svg
paper_authors.fullname: Liang Chen
paper_authors.type: user
paper_authors.user: liangchen-ms
paper_authors._id: 67c17b8f60206395233b7e47
paper_authors.name: Shuai Bai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:33:50.796Z
paper_authors._id: 63451cf0a05b51f7ded25505
paper_authors.avatarUrl: /avatars/dec4bbee4a82b773fc58dfc2dce9dbeb.svg
paper_authors.fullname: shuai bai
paper_authors.type: user
paper_authors.user: bluelike
paper_authors._id: 67c17b8f60206395233b7e48
paper_authors.name: Wenhao Chai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:33:57.358Z
paper_authors._id: 637c7503fe115289cfecbe6b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1676361945047-637c7503fe115289cfecbe6b.jpeg
paper_authors.fullname: Wenhao Chai
paper_authors.type: user
paper_authors.user: wchai
paper_authors._id: 67c17b8f60206395233b7e49
paper_authors.name: Weichu Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:34:48.933Z
paper_authors._id: 678609789a285d232ee14157
paper_authors.avatarUrl: /avatars/a6cb2c571d9ef6deb0b1659f754afe7f.svg
paper_authors.fullname: Weichu Xie
paper_authors.type: user
paper_authors.user: akarinmoe
paper_authors._id: 67c17b8f60206395233b7e4a
paper_authors.name: Haozhe Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:42:16.039Z
paper_authors._id: 63ff09a02098b9ad105a09f6
paper_authors.avatarUrl: /avatars/4409ca5d320050cf4c3df05962c7ff58.svg
paper_authors.fullname: Hans Zhao
paper_authors.type: user
paper_authors.user: BleachNick
paper_authors._id: 67c17b8f60206395233b7e4b
paper_authors.name: Leon Vinci
paper_authors._id: 67c17b8f60206395233b7e4c
paper_authors.name: Junyang Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:34:24.275Z
paper_authors._id: 620760a26e3b7210c2ff1943
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/620760a26e3b7210c2ff1943/VC-rKqimF6yxGESNVlPoR.jpeg
paper_authors.fullname: Junyang Lin
paper_authors.type: user
paper_authors.user: JustinLin610
paper_authors._id: 67c17b8f60206395233b7e4d
paper_authors.name: Baobao Chang
paper_title: Multimodal Representation Alignment for Image Generation: Text-Image
    Interleaved Control Is Easier Than You Think
paper_summary: The field of advanced text-to-image generation is witnessing the emergence of
  unified frameworks that integrate powerful text encoders, such as CLIP and T5,
  with Diffusion Transformer backbones. Although there have been efforts to
  control output images with additional conditions, like canny and depth map, a
  comprehensive framework for arbitrary text-image interleaved control is still
  lacking. This gap is especially evident when attempting to merge concepts or
  visual elements from multiple images in the generation process. To mitigate the
  gap, we conducted preliminary experiments showing that large multimodal models
  (LMMs) offer an effective shared representation space, where image and text can
  be well-aligned to serve as a condition for external diffusion models. Based on
  this discovery, we propose Dream Engine, an efficient and unified framework
  designed for arbitrary text-image interleaved control in image generation
  models. Building on powerful text-to-image models like SD3.5, we replace the
  original text-only encoders by incorporating versatile multimodal information
  encoders such as QwenVL. Our approach utilizes a two-stage training paradigm,
  consisting of joint text-image alignment and multimodal interleaved instruction
  tuning. Our experiments demonstrate that this training method is effective,
  achieving a 0.69 overall score on the GenEval benchmark, and matching the
  performance of state-of-the-art text-to-image models like SD3.5 and FLUX.
paper_ai_summary: A unified framework called Dream Engine integrates powerful text-to-image models with versatile multimodal encoders to enable arbitrary text-image interleaved control in image generation.
paper_ai_keywords: CLIP T5 Diffusion Transformer canny depth map large multimodal models LMMs Dream Engine text-only encoders QwenVL joint text-image alignment multimodal interleaved instruction tuning GenEval benchmark SD3.5 FLUX