Unleashing the Potential of Large Language Models for Text-to-Image
  Generation through Autoregressive Representation Alignment
We present Autoregressive Representation Alignment (ARRA), a new training
framework that unlocks global-coherent text-to-image generation in
autoregressive LLMs without architectural changes. Unlike prior work that
requires complex architectural redesigns, ARRA aligns LLM hidden states with
visual representations from external visual foundational models via a global
visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual
constraints: local next-token prediction and global semantic distillation,
enabling LLMs to implicitly learn spatial and contextual coherence while
retaining their original autoregressive paradigm. Extensive experiments
validate ARRA's plug-and-play versatility. When training from
text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5%
(MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive
LLMs like Chameleon and LlamaGen, all without framework modifications. For
domain adaption, ARRA aligns general-purpose LLMs with specialized models
(e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on
medical imaging (MIMIC-CXR). By demonstrating that training objective redesign
-- not just architectural innovation -- can resolve cross-modal global
coherence challenges, ARRA offers a complementary paradigm for advancing
autoregressive models. Code and models will be released to advance
autoregressive image generation.
67d04cecf9bcb732f0880c96
Xing Xie
claimed_verified
2025-03-11T14:55:23.006Z
67887a429ede23ff0a964c98
/avatars/12dea48e4b37e8a7f5c9bbbc733f2348.svg
xing xie
user
xing0916
67d04cecf9bcb732f0880c97
Jiawei Liu
claimed_verified
2025-03-12T08:42:09.450Z
6440f16a5d600fb095198f15
https://cdn-avatars.huggingface.co/v1/production/uploads/6440f16a5d600fb095198f15/-J89Lelh3PPRVcc1ya_VS.png
Jiawei Liu
user
nachifur
67d04cecf9bcb732f0880c98
Ziyue Lin
67d04cecf9bcb732f0880c99
Huijie Fan
67d04cecf9bcb732f0880c9a
Zhi Han
67d04cecf9bcb732f0880c9b
Yandong Tang
67d04cecf9bcb732f0880c9c
Liangqiong Qu
Unleashing the Potential of Large Language Models for Text-to-Image
  Generation through Autoregressive Representation Alignment
We present Autoregressive Representation Alignment (ARRA), a new training
framework that unlocks global-coherent text-to-image generation in
autoregressive LLMs without architectural changes. Unlike prior work that
requires complex architectural redesigns, ARRA aligns LLM hidden states with
visual representations from external visual foundational models via a global
visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual
constraints: local next-token prediction and global semantic distillation,
enabling LLMs to implicitly learn spatial and contextual coherence while
retaining their original autoregressive paradigm. Extensive experiments
validate ARRA's plug-and-play versatility. When training from
text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5%
(MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive
LLMs like Chameleon and LlamaGen, all without framework modifications. For
domain adaption, ARRA aligns general-purpose LLMs with specialized models
(e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on
medical imaging (MIMIC-CXR). By demonstrating that training objective redesign
-- not just architectural innovation -- can resolve cross-modal global
coherence challenges, ARRA offers a complementary paradigm for advancing
autoregressive models. Code and models will be released to advance
autoregressive image generation.
ARRA framework enables global-coherent text-to-image generation in autoregressive LLMs by aligning hidden states with visual representations via a global alignment loss and hybrid token.
Autoregressive Representation Alignment (ARRA) autoregressive LLMs global-coherent text-to-image generation global visual alignment loss hybrid token local next-token prediction global semantic distillation FID MIMIC-CXR DeepEyeNet ImageNet Chameleon LlamaGen domain adaption BioMedCLIP