title: ControlAR: Controllable Image Generation with Autoregressive Models
summary: Autoregressive (AR) models have reformulated image generation as next-token
  prediction, demonstrating remarkable potential and emerging as strong
  competitors to diffusion models. However, control-to-image generation, akin to
  ControlNet, remains largely unexplored within AR models. Although a natural
  approach, inspired by advancements in Large Language Models, is to tokenize
  control images into tokens and prefill them into the autoregressive model
  before decoding image tokens, it still falls short in generation quality
  compared to ControlNet and suffers from inefficiency. To this end, we introduce
  ControlAR, an efficient and effective framework for integrating spatial
  controls into autoregressive image generation models. Firstly, we explore
  control encoding for AR models and propose a lightweight control encoder to
  transform spatial inputs (e.g., canny edges or depth maps) into control tokens.
  Then ControlAR exploits the conditional decoding method to generate the next
  image token conditioned on the per-token fusion between control and image
  tokens, similar to positional encodings. Compared to prefilling tokens, using
  conditional decoding significantly strengthens the control capability of AR
  models but also maintains the model's efficiency. Furthermore, the proposed
  ControlAR surprisingly empowers AR models with arbitrary-resolution image
  generation via conditional decoding and specific controls. Extensive
  experiments can demonstrate the controllability of the proposed ControlAR for
  the autoregressive control-to-image generation across diverse inputs, including
  edges, depths, and segmentation masks. Furthermore, both quantitative and
  qualitative results indicate that ControlAR surpasses previous state-of-the-art
  controllable diffusion models, e.g., ControlNet++. Code, models, and demo will
  soon be available at https://github.com/hustvl/ControlAR.
paper_authors._id: 670635fa5178a313ef69ba21
paper_authors.name: Zongming Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-04T08:53:31.237Z
paper_authors._id: 653b39ba170f0a2a92102657
paper_authors.avatarUrl: /avatars/b05bfbce5f3e4c44655cb6672f372c70.svg
paper_authors.fullname: Zongming Li
paper_authors.type: user
paper_authors.user: zongmingli
paper_authors._id: 670635fa5178a313ef69ba22
paper_authors.name: Tianheng Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:54:40.586Z
paper_authors._id: 646b3db131968a60a01e4cf5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
paper_authors.fullname: Tianheng Cheng
paper_authors.type: user
paper_authors.user: wondervictor
paper_authors._id: 670635fa5178a313ef69ba23
paper_authors.name: Shoufa Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:54:47.669Z
paper_authors._id: 6412a33900634c4fe9873652
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
paper_authors.fullname: Shoufa Chen
paper_authors.type: user
paper_authors.user: ShoufaChen
paper_authors._id: 670635fa5178a313ef69ba24
paper_authors.name: Peize Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:54:55.199Z
paper_authors._id: 640dc9bf8512ec51d7f0ac1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
paper_authors.fullname: peizesun
paper_authors.type: user
paper_authors.user: peizesun
paper_authors._id: 670635fa5178a313ef69ba25
paper_authors.name: Haocheng Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:55:02.914Z
paper_authors._id: 66504b391ca90edc544280d2
paper_authors.avatarUrl: /avatars/81ac0cab23802346ae154ca5918f64cc.svg
paper_authors.fullname: Haocheng Shen
paper_authors.type: user
paper_authors.user: HCKodiak
paper_authors._id: 670635fa5178a313ef69ba26
paper_authors.name: Longjin Ran
paper_authors._id: 670635fa5178a313ef69ba27
paper_authors.name: Xiaoxin Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:55:23.910Z
paper_authors._id: 65389a669c474315d7425f96
paper_authors.avatarUrl: /avatars/2fa3828ca489cfe1948129a0eccf264f.svg
paper_authors.fullname: chenxiaoxin
paper_authors.type: user
paper_authors.user: steelozazala
paper_authors._id: 670635fa5178a313ef69ba28
paper_authors.name: Wenyu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:55:29.707Z
paper_authors._id: 66c2e7fc934e2f07753542ac
paper_authors.avatarUrl: /avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
paper_authors.fullname: LWY
paper_authors.type: user
paper_authors.user: wenyuliu
paper_authors._id: 670635fa5178a313ef69ba29
paper_authors.name: Xinggang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-09T12:55:35.614Z
paper_authors._id: 62600de6d47e3dbae32ce1ce
paper_authors.avatarUrl: /avatars/a536417cfec6e10ac415091bd1829426.svg
paper_authors.fullname: Xinggang Wang
paper_authors.type: user
paper_authors.user: xinggangw
paper_title: ControlAR: Controllable Image Generation with Autoregressive Models
paper_summary: Autoregressive (AR) models have reformulated image generation as next-token
  prediction, demonstrating remarkable potential and emerging as strong
  competitors to diffusion models. However, control-to-image generation, akin to
  ControlNet, remains largely unexplored within AR models. Although a natural
  approach, inspired by advancements in Large Language Models, is to tokenize
  control images into tokens and prefill them into the autoregressive model
  before decoding image tokens, it still falls short in generation quality
  compared to ControlNet and suffers from inefficiency. To this end, we introduce
  ControlAR, an efficient and effective framework for integrating spatial
  controls into autoregressive image generation models. Firstly, we explore
  control encoding for AR models and propose a lightweight control encoder to
  transform spatial inputs (e.g., canny edges or depth maps) into control tokens.
  Then ControlAR exploits the conditional decoding method to generate the next
  image token conditioned on the per-token fusion between control and image
  tokens, similar to positional encodings. Compared to prefilling tokens, using
  conditional decoding significantly strengthens the control capability of AR
  models but also maintains the model's efficiency. Furthermore, the proposed
  ControlAR surprisingly empowers AR models with arbitrary-resolution image
  generation via conditional decoding and specific controls. Extensive
  experiments can demonstrate the controllability of the proposed ControlAR for
  the autoregressive control-to-image generation across diverse inputs, including
  edges, depths, and segmentation masks. Furthermore, both quantitative and
  qualitative results indicate that ControlAR surpasses previous state-of-the-art
  controllable diffusion models, e.g., ControlNet++. Code, models, and demo will
  soon be available at https://github.com/hustvl/ControlAR.
paper_ai_keywords: autoregressive (AR) models next-token prediction diffusion models control-to-image generation ControlNet tokenization pretrained autoencoders image tokens ControlAR spatial inputs canny edges depth maps control tokens conditional decoding positional encodings arbitrary-resolution image generation specific controls controllability