ControlAR: Controllable Image Generation with Autoregressive Models
Autoregressive (AR) models have reformulated image generation as next-token
prediction, demonstrating remarkable potential and emerging as strong
competitors to diffusion models. However, control-to-image generation, akin to
ControlNet, remains largely unexplored within AR models. Although a natural
approach, inspired by advancements in Large Language Models, is to tokenize
control images into tokens and prefill them into the autoregressive model
before decoding image tokens, it still falls short in generation quality
compared to ControlNet and suffers from inefficiency. To this end, we introduce
ControlAR, an efficient and effective framework for integrating spatial
controls into autoregressive image generation models. Firstly, we explore
control encoding for AR models and propose a lightweight control encoder to
transform spatial inputs (e.g., canny edges or depth maps) into control tokens.
Then ControlAR exploits the conditional decoding method to generate the next
image token conditioned on the per-token fusion between control and image
tokens, similar to positional encodings. Compared to prefilling tokens, using
conditional decoding significantly strengthens the control capability of AR
models but also maintains the model's efficiency. Furthermore, the proposed
ControlAR surprisingly empowers AR models with arbitrary-resolution image
generation via conditional decoding and specific controls. Extensive
experiments can demonstrate the controllability of the proposed ControlAR for
the autoregressive control-to-image generation across diverse inputs, including
edges, depths, and segmentation masks. Furthermore, both quantitative and
qualitative results indicate that ControlAR surpasses previous state-of-the-art
controllable diffusion models, e.g., ControlNet++. Code, models, and demo will
soon be available at https://github.com/hustvl/ControlAR.
670635fa5178a313ef69ba21
Zongming Li
claimed_verified
2025-03-04T08:53:31.237Z
653b39ba170f0a2a92102657
/avatars/b05bfbce5f3e4c44655cb6672f372c70.svg
Zongming Li
user
zongmingli
670635fa5178a313ef69ba22
Tianheng Cheng
admin_assigned
2024-10-09T12:54:40.586Z
646b3db131968a60a01e4cf5
https://cdn-avatars.huggingface.co/v1/production/uploads/646b3db131968a60a01e4cf5/DhfdqUYQaD1Qa8Svw996J.jpeg
Tianheng Cheng
user
wondervictor
670635fa5178a313ef69ba23
Shoufa Chen
admin_assigned
2024-10-09T12:54:47.669Z
6412a33900634c4fe9873652
https://cdn-avatars.huggingface.co/v1/production/uploads/6412a33900634c4fe9873652/Nmn_yRA1gGD2VO1YbSOYF.jpeg
Shoufa Chen
user
ShoufaChen
670635fa5178a313ef69ba24
Peize Sun
admin_assigned
2024-10-09T12:54:55.199Z
640dc9bf8512ec51d7f0ac1a
https://cdn-avatars.huggingface.co/v1/production/uploads/640dc9bf8512ec51d7f0ac1a/sT4rdEoQbzfW6D3xDVdqt.jpeg
peizesun
user
peizesun
670635fa5178a313ef69ba25
Haocheng Shen
admin_assigned
2024-10-09T12:55:02.914Z
66504b391ca90edc544280d2
/avatars/81ac0cab23802346ae154ca5918f64cc.svg
Haocheng Shen
user
HCKodiak
670635fa5178a313ef69ba26
Longjin Ran
670635fa5178a313ef69ba27
Xiaoxin Chen
admin_assigned
2024-10-09T12:55:23.910Z
65389a669c474315d7425f96
/avatars/2fa3828ca489cfe1948129a0eccf264f.svg
chenxiaoxin
user
steelozazala
670635fa5178a313ef69ba28
Wenyu Liu
admin_assigned
2024-10-09T12:55:29.707Z
66c2e7fc934e2f07753542ac
/avatars/f6fa3f94435cf1c1d06daa6c925d07d0.svg
LWY
user
wenyuliu
670635fa5178a313ef69ba29
Xinggang Wang
admin_assigned
2024-10-09T12:55:35.614Z
62600de6d47e3dbae32ce1ce
/avatars/a536417cfec6e10ac415091bd1829426.svg
Xinggang Wang
user
xinggangw
ControlAR: Controllable Image Generation with Autoregressive Models
Autoregressive (AR) models have reformulated image generation as next-token
prediction, demonstrating remarkable potential and emerging as strong
competitors to diffusion models. However, control-to-image generation, akin to
ControlNet, remains largely unexplored within AR models. Although a natural
approach, inspired by advancements in Large Language Models, is to tokenize
control images into tokens and prefill them into the autoregressive model
before decoding image tokens, it still falls short in generation quality
compared to ControlNet and suffers from inefficiency. To this end, we introduce
ControlAR, an efficient and effective framework for integrating spatial
controls into autoregressive image generation models. Firstly, we explore
control encoding for AR models and propose a lightweight control encoder to
transform spatial inputs (e.g., canny edges or depth maps) into control tokens.
Then ControlAR exploits the conditional decoding method to generate the next
image token conditioned on the per-token fusion between control and image
tokens, similar to positional encodings. Compared to prefilling tokens, using
conditional decoding significantly strengthens the control capability of AR
models but also maintains the model's efficiency. Furthermore, the proposed
ControlAR surprisingly empowers AR models with arbitrary-resolution image
generation via conditional decoding and specific controls. Extensive
experiments can demonstrate the controllability of the proposed ControlAR for
the autoregressive control-to-image generation across diverse inputs, including
edges, depths, and segmentation masks. Furthermore, both quantitative and
qualitative results indicate that ControlAR surpasses previous state-of-the-art
controllable diffusion models, e.g., ControlNet++. Code, models, and demo will
soon be available at https://github.com/hustvl/ControlAR.
ControlAR enhances autoregressive image generation with spatial controls, offering higher quality and arbitrary-resolution output compared to existing models.
autoregressive models next-token prediction diffusion models ControlNet control images tokens lightweight control encoder control tokens conditional decoding positional encodings arbitrary-resolution image generation