I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning
  in Diffusion Models
This paper presents ThinkDiff, a novel alignment paradigm that empowers
text-to-image diffusion models with multimodal in-context understanding and
reasoning capabilities by integrating the strengths of vision-language models
(VLMs). Existing multimodal diffusion finetuning methods largely focus on
pixel-level reconstruction rather than in-context reasoning, and are
constrained by the complexity and limited availability of reasoning-based
datasets. ThinkDiff addresses these challenges by leveraging vision-language
training as a proxy task, aligning VLMs with the decoder of an encoder-decoder
large language model (LLM) instead of a diffusion decoder. This proxy task
builds on the observation that the LLM decoder shares the same input
feature space with diffusion decoders that use the corresponding
LLM encoder for prompt embedding. As a result, aligning VLMs with
diffusion decoders can be simplified through alignment with the LLM decoder.
Without complex training and datasets, ThinkDiff effectively unleashes
understanding, reasoning, and composing capabilities in diffusion models.
Experiments demonstrate that ThinkDiff significantly improves accuracy from
19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context
reasoning generation, with only 5 hours of training on 4 A100 GPUs.
Additionally, ThinkDiff demonstrates exceptional performance in composing
multiple images and texts into logically coherent images. Project page:
https://mizhenxing.github.io/ThinkDiff.
67b3ea0f4dd7ea0538ce589d
Zhenxing Mi
claimed_verified
2025-02-18T09:31:52.837Z
6354bda206d707b33249c4c2
/avatars/bbd9f76274ac52214df92084d50bc7b5.svg
Zhenxing Mi
user
Mifucius
67b3ea0f4dd7ea0538ce589e
Kuan-Chieh Wang
admin_assigned
2025-02-19T15:21:46.349Z
648ca58a39d2584ee47efef6
https://cdn-avatars.huggingface.co/v1/production/uploads/648ca58a39d2584ee47efef6/R7B72bnwc59mdK45rmzYS.png
Kuan-Chieh Wang
user
wangkua1
67b3ea0f4dd7ea0538ce589f
Guocheng Qian
admin_assigned
2025-02-19T15:21:52.861Z
645fed74335c21d19f3bf76c
https://cdn-avatars.huggingface.co/v1/production/uploads/645fed74335c21d19f3bf76c/gwVsllRWtSHbg4a1erkdF.jpeg
Guocheng Qian
user
guochengqian
67b3ea0f4dd7ea0538ce58a0
Hanrong Ye
admin_assigned
2025-02-19T15:21:58.865Z
62d3ae4d894e7fe42def988f
/avatars/3aafc55d9783459f9a79546fc31dd68a.svg
Hanrong Ye
user
leoye
67b3ea0f4dd7ea0538ce58a1
Runtao Liu
admin_assigned
2025-02-19T15:22:08.197Z
64a653e330dd11336539c439
/avatars/348910ea160829707ac5e74f9f824c60.svg
liuruntao
user
runtao
67b3ea0f4dd7ea0538ce58a2
Sergey Tulyakov
67b3ea0f4dd7ea0538ce58a3
Kfir Aberman
admin_assigned
2025-02-19T15:22:17.444Z
64db29097266618e853dd6ec
https://cdn-avatars.huggingface.co/v1/production/uploads/64db29097266618e853dd6ec/r0MaPQCfAxeKv3ycdKYLK.jpeg
Kfir Aberman
user
kaberman
67b3ea0f4dd7ea0538ce58a4
Dan Xu
claimed_verified
2025-02-19T09:04:53.095Z
66feab48651e00e22f33222e
/avatars/7344377e2c796c7ec85194bb2fc78521.svg
Dan Xu
user
danxuhk
I Think, Therefore I Diffuse: Enabling Multimodal In-Context Reasoning
  in Diffusion Models
This paper presents ThinkDiff, a novel alignment paradigm that empowers
text-to-image diffusion models with multimodal in-context understanding and
reasoning capabilities by integrating the strengths of vision-language models
(VLMs). Existing multimodal diffusion finetuning methods largely focus on
pixel-level reconstruction rather than in-context reasoning, and are
constrained by the complexity and limited availability of reasoning-based
datasets. ThinkDiff addresses these challenges by leveraging vision-language
training as a proxy task, aligning VLMs with the decoder of an encoder-decoder
large language model (LLM) instead of a diffusion decoder. This proxy task
builds on the observation that the LLM decoder shares the same input
feature space with diffusion decoders that use the corresponding
LLM encoder for prompt embedding. As a result, aligning VLMs with
diffusion decoders can be simplified through alignment with the LLM decoder.
Without complex training and datasets, ThinkDiff effectively unleashes
understanding, reasoning, and composing capabilities in diffusion models.
Experiments demonstrate that ThinkDiff significantly improves accuracy from
19.2% to 46.3% on the challenging CoBSAT benchmark for multimodal in-context
reasoning generation, with only 5 hours of training on 4 A100 GPUs.
Additionally, ThinkDiff demonstrates exceptional performance in composing
multiple images and texts into logically coherent images. Project page:
https://mizhenxing.github.io/ThinkDiff.
ThinkDiff improves multimodal reasoning in text-to-image diffusion models using vision-language models aligned with the decoder of an encoder-decoder large language model.
vision-language models multimodal diffusion in-context reasoning encoder-decoder.large language model diffusion decoders LLM decoder CoBSAT benchmark logically coherent images