title: I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning
    in Diffusion Models
summary: This paper presents ThinkDiff, a novel alignment paradigm that empowers
  text-to-image diffusion models with multimodal in-context understanding and
  reasoning capabilities by integrating the strengths of vision-language models
  (VLMs). Existing multimodal diffusion finetuning methods largely focus on
  pixel-level reconstruction rather than in-context reasoning, and are
  constrained by the complexity and limited availability of reasoning-based
  datasets. ThinkDiff addresses these challenges by leveraging vision-language
  training as a proxy task, aligning VLMs with the decoder of an encoder-decoder
  large language model (LLM) instead of a diffusion decoder. This proxy task
  builds on the observation that the LLM decoder shares the same input
  feature space with diffusion decoders that use the corresponding
  LLM encoder for prompt embedding. As a result, aligning VLMs with
  diffusion decoders can be simplified through alignment with the LLM decoder.
  Without complex training and datasets, ThinkDiff effectively unleashes
  understanding, reasoning, and composing capabilities in diffusion models.
  Experiments demonstrate that ThinkDiff significantly improves accuracy from
  19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context
  reasoning generation, with only 5 hours of training on 4 A100 GPUs.
  Additionally, ThinkDiff demonstrates exceptional performance in composing
  multiple images and texts into logically coherent images. Project page:
  https://mizhenxing.github.io/ThinkDiff.
paper_authors._id: 67b3ea0f4dd7ea0538ce589d
paper_authors.name: Zhenxing Mi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:31:52.837Z
paper_authors._id: 6354bda206d707b33249c4c2
paper_authors.avatarUrl: /avatars/bbd9f76274ac52214df92084d50bc7b5.svg
paper_authors.fullname: Zhenxing Mi
paper_authors.type: user
paper_authors.user: Mifucius
paper_authors._id: 67b3ea0f4dd7ea0538ce589e
paper_authors.name: Kuan-Chieh Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:21:46.349Z
paper_authors._id: 648ca58a39d2584ee47efef6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
paper_authors.fullname: Kuan-Chieh Wang
paper_authors.type: user
paper_authors.user: wangkua1
paper_authors._id: 67b3ea0f4dd7ea0538ce589f
paper_authors.name: Guocheng Qian
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:21:52.861Z
paper_authors._id: 645fed74335c21d19f3bf76c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/645fed74335c21d19f3bf76c/gwVsllRWtSHbg4a1erkdF.jpeg
paper_authors.fullname: Guocheng Qian
paper_authors.type: user
paper_authors.user: guochengqian
paper_authors._id: 67b3ea0f4dd7ea0538ce58a0
paper_authors.name: Hanrong Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:21:58.865Z
paper_authors._id: 62d3ae4d894e7fe42def988f
paper_authors.avatarUrl: /avatars/3aafc55d9783459f9a79546fc31dd68a.svg
paper_authors.fullname: Hanrong Ye
paper_authors.type: user
paper_authors.user: leoye
paper_authors._id: 67b3ea0f4dd7ea0538ce58a1
paper_authors.name: Runtao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:22:08.197Z
paper_authors._id: 64a653e330dd11336539c439
paper_authors.avatarUrl: /avatars/348910ea160829707ac5e74f9f824c60.svg
paper_authors.fullname: liuruntao
paper_authors.type: user
paper_authors.user: runtao
paper_authors._id: 67b3ea0f4dd7ea0538ce58a2
paper_authors.name: Sergey Tulyakov
paper_authors._id: 67b3ea0f4dd7ea0538ce58a3
paper_authors.name: Kfir Aberman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-19T15:22:17.444Z
paper_authors._id: 64db29097266618e853dd6ec
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
paper_authors.fullname: Kfir Aberman
paper_authors.type: user
paper_authors.user: kaberman
paper_authors._id: 67b3ea0f4dd7ea0538ce58a4
paper_authors.name: Dan Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-19T09:04:53.095Z
paper_authors._id: 66feab48651e00e22f33222e
paper_authors.avatarUrl: /avatars/7344377e2c796c7ec85194bb2fc78521.svg
paper_authors.fullname: Dan Xu
paper_authors.type: user
paper_authors.user: danxuhk
paper_title: I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning
    in Diffusion Models
paper_summary: This paper presents ThinkDiff, a novel alignment paradigm that empowers
  text-to-image diffusion models with multimodal in-context understanding and
  reasoning capabilities by integrating the strengths of vision-language models
  (VLMs). Existing multimodal diffusion finetuning methods largely focus on
  pixel-level reconstruction rather than in-context reasoning, and are
  constrained by the complexity and limited availability of reasoning-based
  datasets. ThinkDiff addresses these challenges by leveraging vision-language
  training as a proxy task, aligning VLMs with the decoder of an encoder-decoder
  large language model (LLM) instead of a diffusion decoder. This proxy task
  builds on the observation that the LLM decoder shares the same input
  feature space with diffusion decoders that use the corresponding
  LLM encoder for prompt embedding. As a result, aligning VLMs with
  diffusion decoders can be simplified through alignment with the LLM decoder.
  Without complex training and datasets, ThinkDiff effectively unleashes
  understanding, reasoning, and composing capabilities in diffusion models.
  Experiments demonstrate that ThinkDiff significantly improves accuracy from
  19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context
  reasoning generation, with only 5 hours of training on 4 A100 GPUs.
  Additionally, ThinkDiff demonstrates exceptional performance in composing
  multiple images and texts into logically coherent images. Project page:
  https://mizhenxing.github.io/ThinkDiff.
paper_ai_keywords: ThinkDiff alignment paradigm text-to-image diffusion models multimodal in-context understanding reasoning capabilities Vision-language models (VLMs) pixel focud multimodal diffusion finetuning methods pixel-level reconstruction in-context expectation reasoning-based datasets vision-language training proxy task LLM decoders diffusion decoders multimodal encoder prompt embedding multimodal in-context reasoning generation CoBSAT benchmark