Towards Multimodal Understanding via Stable Diffusion as a Task-Aware
  Feature Extractor
Recent advances in multimodal large language models (MLLMs) have enabled
image-based question-answering capabilities. However, a key limitation is the
use of CLIP as the visual encoder; while it can capture coarse global
information, it often can miss fine-grained details that are relevant to the
input query. To address these shortcomings, this work studies whether
pre-trained text-to-image diffusion models can serve as instruction-aware
visual encoders. Through an analysis of their internal representations, we find
diffusion features are both rich in semantics and can encode strong image-text
alignment. Moreover, we find that we can leverage text conditioning to focus
the model on regions relevant to the input question. We then investigate how to
align these features with large language models and uncover a leakage
phenomenon, where the LLM can inadvertently recover information from the
original diffusion prompt. We analyze the causes of this leakage and propose a
mitigation strategy. Based on these insights, we explore a simple fusion
strategy that utilizes both CLIP and conditional diffusion features. We
evaluate our approach on both general VQA and specialized MLLM benchmarks,
demonstrating the promise of diffusion models for visual understanding,
particularly in vision-centric tasks that require spatial and compositional
reasoning. Our project page can be found
https://vatsalag99.github.io/mustafar/.
686f4265d938c25d68441ba4
Vatsal Agarwal
claimed_verified
2025-07-10T07:09:01.062Z
653213eaa0dffcea236ca422
/avatars/69d30ce923edcf99e177669c2f8bf9a5.svg
Vatsal Agarwal
user
vatsalag
686f4265d938c25d68441ba5
Matthew Gwilliam
686f4265d938c25d68441ba6
Gefen Kohavi
686f4265d938c25d68441ba7
Eshan Verma
686f4265d938c25d68441ba8
Daniel Ulbricht
686f4265d938c25d68441ba9
Abhinav Shrivastava
Towards Multimodal Understanding via Stable Diffusion as a Task-Aware
  Feature Extractor
Recent advances in multimodal large language models (MLLMs) have enabled
image-based question-answering capabilities. However, a key limitation is the
use of CLIP as the visual encoder; while it can capture coarse global
information, it often can miss fine-grained details that are relevant to the
input query. To address these shortcomings, this work studies whether
pre-trained text-to-image diffusion models can serve as instruction-aware
visual encoders. Through an analysis of their internal representations, we find
diffusion features are both rich in semantics and can encode strong image-text
alignment. Moreover, we find that we can leverage text conditioning to focus
the model on regions relevant to the input question. We then investigate how to
align these features with large language models and uncover a leakage
phenomenon, where the LLM can inadvertently recover information from the
original diffusion prompt. We analyze the causes of this leakage and propose a
mitigation strategy. Based on these insights, we explore a simple fusion
strategy that utilizes both CLIP and conditional diffusion features. We
evaluate our approach on both general VQA and specialized MLLM benchmarks,
demonstrating the promise of diffusion models for visual understanding,
particularly in vision-centric tasks that require spatial and compositional
reasoning. Our project page can be found
https://vatsalag99.github.io/mustafar/.
Pre-trained text-to-image diffusion models enhance image-based question-answering by providing rich semantic features and strong image-text alignment, while a fusion strategy with CLIP addresses leakage issues.
multimodal large language models MLLMs image-based question-answering CLIP visual encoder diffusion models instruction-aware internal representations semantics image-text alignment text conditioning leakage phenomenon fusion strategy VQA vision-centric tasks spatial reasoning compositional reasoning