title: Unleashing the Potential of Large Language Models for Text-to-Image
    Generation through Autoregressive Representation Alignment
summary: We present Autoregressive Representation Alignment (ARRA), a new training
  framework that unlocks global-coherent text-to-image generation in
  autoregressive LLMs without architectural changes. Unlike prior work that
  requires complex architectural redesigns, ARRA aligns LLM hidden states with
  visual representations from external visual foundational models via a global
  visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual
  constraints: local next-token prediction and global semantic distillation,
  enabling LLMs to implicitly learn spatial and contextual coherence while
  retaining their original autoregressive paradigm. Extensive experiments
  validate ARRA's plug-and-play versatility. When training from
  text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5%
  (MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive
  LLMs like Chameleon and LlamaGen, all without framework modifications. For
  domain adaption, ARRA aligns general-purpose LLMs with specialized models
  (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on
  medical imaging (MIMIC-CXR). By demonstrating that training objective redesign
  -- not just architectural innovation -- can resolve cross-modal global
  coherence challenges, ARRA offers a complementary paradigm for advancing
  autoregressive models. Code and models will be released to advance
  autoregressive image generation.
paper_authors._id: 67d04cecf9bcb732f0880c96
paper_authors.name: Xing Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T14:55:23.006Z
paper_authors._id: 67887a429ede23ff0a964c98
paper_authors.avatarUrl: /avatars/12dea48e4b37e8a7f5c9bbbc733f2348.svg
paper_authors.fullname: xing xie
paper_authors.type: user
paper_authors.user: xing0916
paper_authors._id: 67d04cecf9bcb732f0880c97
paper_authors.name: Jiawei Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-12T08:42:09.450Z
paper_authors._id: 6440f16a5d600fb095198f15
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6440f16a5d600fb095198f15/-J89Lelh3PPRVcc1ya_VS.png
paper_authors.fullname: Jiawei Liu
paper_authors.type: user
paper_authors.user: nachifur
paper_authors._id: 67d04cecf9bcb732f0880c98
paper_authors.name: Ziyue Lin
paper_authors._id: 67d04cecf9bcb732f0880c99
paper_authors.name: Huijie Fan
paper_authors._id: 67d04cecf9bcb732f0880c9a
paper_authors.name: Zhi Han
paper_authors._id: 67d04cecf9bcb732f0880c9b
paper_authors.name: Yandong Tang
paper_authors._id: 67d04cecf9bcb732f0880c9c
paper_authors.name: Liangqiong Qu
paper_title: Unleashing the Potential of Large Language Models for Text-to-Image
    Generation through Autoregressive Representation Alignment
paper_summary: We present Autoregressive Representation Alignment (ARRA), a new training
  framework that unlocks global-coherent text-to-image generation in
  autoregressive LLMs without architectural changes. Unlike prior work that
  requires complex architectural redesigns, ARRA aligns LLM hidden states with
  visual representations from external visual foundational models via a global
  visual alignment loss and a hybrid token, <HYBNEXT>. This token enforces dual
  constraints: local next-token prediction and global semantic distillation,
  enabling LLMs to implicitly learn spatial and contextual coherence while
  retaining their original autoregressive paradigm. Extensive experiments
  validate ARRA's plug-and-play versatility. When training from
  text-generation-only LLMs or random initialization, ARRA reduces FID by 25.5%
  (MIMIC-CXR), 8.8% (DeepEyeNet), and 7.5% (ImageNet) for advanced autoregressive
  LLMs like Chameleon and LlamaGen, all without framework modifications. For
  domain adaption, ARRA aligns general-purpose LLMs with specialized models
  (e.g., BioMedCLIP), achieving an 18.6% FID reduction over direct fine-tuning on
  medical imaging (MIMIC-CXR). By demonstrating that training objective redesign
  -- not just architectural innovation -- can resolve cross-modal global
  coherence challenges, ARRA offers a complementary paradigm for advancing
  autoregressive models. Code and models will be released to advance
  autoregressive image generation.
paper_ai_summary: ARRA framework enables global-coherent text-to-image generation in autoregressive LLMs by aligning hidden states with visual representations via a global alignment loss and hybrid token.
paper_ai_keywords: Autoregressive Representation Alignment (ARRA) autoregressive LLMs global-coherent text-to-image generation global visual alignment loss hybrid token local next-token prediction global semantic distillation FID MIMIC-CXR DeepEyeNet ImageNet Chameleon LlamaGen domain adaption BioMedCLIP