Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation
  from Diffusion Models
Building state-of-the-art Vision-Language Models (VLMs) with strong
captioning capabilities typically necessitates training on billions of
high-quality image-text pairs, requiring millions of GPU hours. This paper
introduces the Vision-Language-Vision (VLV) auto-encoder framework, which
strategically leverages key pretrained components: a vision encoder, the
decoder of a Text-to-Image (T2I) diffusion model, and subsequently, a Large
Language Model (LLM). Specifically, we establish an information bottleneck by
regularizing the language representation space, achieved through freezing the
pretrained T2I diffusion decoder. Our VLV pipeline effectively distills
knowledge from the text-conditioned diffusion model using continuous
embeddings, demonstrating comprehensive semantic understanding via high-quality
reconstructions. Furthermore, by fine-tuning a pretrained LLM to decode the
intermediate language representations into detailed descriptions, we construct
a state-of-the-art (SoTA) captioner comparable to leading models like GPT-4o
and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and
significantly reduces data requirements; by primarily utilizing single-modal
images for training and maximizing the utility of existing pretrained models
(image encoder, T2I diffusion model, and LLM), it circumvents the need for
massive paired image-text datasets, keeping the total training expenditure
under $1,000 USD.
686f95e9706a6ea4654189ff
Tiezheng Zhang
claimed_verified
2025-07-11T08:03:27.231Z
66e0b013733965882099cc37
https://cdn-avatars.huggingface.co/v1/production/uploads/66e0b013733965882099cc37/CkTK2kV2v-TfdYiwsW6Tx.jpeg
Tiezheng Zhang
user
PatZhang11
686f95e9706a6ea465418a00
Yitong Li
686f95e9706a6ea465418a01
Yu-cheng Chou
686f95e9706a6ea465418a02
Jieneng Chen
686f95e9706a6ea465418a03
Alan Yuille
686f95e9706a6ea465418a04
Chen Wei
686f95e9706a6ea465418a05
Junfei Xiao
claimed_verified
2025-07-11T08:03:25.003Z
64b5ba6060274cbb296d6288
/avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
Junfei Xiao
user
lambertxiao
Vision-Language-Vision Auto-Encoder: Scalable Knowledge Distillation
  from Diffusion Models
Building state-of-the-art Vision-Language Models (VLMs) with strong
captioning capabilities typically necessitates training on billions of
high-quality image-text pairs, requiring millions of GPU hours. This paper
introduces the Vision-Language-Vision (VLV) auto-encoder framework, which
strategically leverages key pretrained components: a vision encoder, the
decoder of a Text-to-Image (T2I) diffusion model, and subsequently, a Large
Language Model (LLM). Specifically, we establish an information bottleneck by
regularizing the language representation space, achieved through freezing the
pretrained T2I diffusion decoder. Our VLV pipeline effectively distills
knowledge from the text-conditioned diffusion model using continuous
embeddings, demonstrating comprehensive semantic understanding via high-quality
reconstructions. Furthermore, by fine-tuning a pretrained LLM to decode the
intermediate language representations into detailed descriptions, we construct
a state-of-the-art (SoTA) captioner comparable to leading models like GPT-4o
and Gemini 2.0 Flash. Our method demonstrates exceptional cost-efficiency and
significantly reduces data requirements; by primarily utilizing single-modal
images for training and maximizing the utility of existing pretrained models
(image encoder, T2I diffusion model, and LLM), it circumvents the need for
massive paired image-text datasets, keeping the total training expenditure
under $1,000 USD.
The VLV auto-encoder framework uses pretrained vision and text models to create a cost-efficient, high-quality captioning system with reduced data requirements.
Vision-Language Models VLV auto-encoder vision encoder Text-to-Image diffusion model Large Language Model information bottleneck language representation space continuous embeddings semantic understanding high-quality reconstructions fine-tuning captioner GPT-4o Gemini 2.0 Flash