title: Autoregressive Models in Vision: A Survey
summary: Autoregressive modeling has been a huge success in the field of natural
  language processing (NLP). Recently, autoregressive models have emerged as a
  significant area of focus in computer vision, where they excel in producing
  high-quality visual content. Autoregressive models in NLP typically operate on
  subword tokens. However, the representation strategy in computer vision can
  vary in different levels, i.e., pixel-level, token-level, or
  scale-level, reflecting the diverse and hierarchical nature of visual data
  compared to the sequential structure of language. This survey comprehensively
  examines the literature on autoregressive models applied to vision. To improve
  readability for researchers from diverse research backgrounds, we start with
  preliminary sequence representation and modeling in vision. Next, we divide the
  fundamental frameworks of visual autoregressive models into three general
  sub-categories, including pixel-based, token-based, and scale-based models
  based on the strategy of representation. We then explore the interconnections
  between autoregressive models and other generative models. Furthermore, we
  present a multi-faceted categorization of autoregressive models in computer
  vision, including image generation, video generation, 3D generation, and
  multi-modal generation. We also elaborate on their applications in diverse
  domains, including emerging domains such as embodied AI and 3D medical AI, with
  about 250 related references. Finally, we highlight the current challenges to
  autoregressive models in vision with suggestions about potential research
  directions. We have also set up a Github repository to organize the papers
  included in this survey at:
  https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.
paper_authors._id: 673395a92f0f0d8e0ec5af19
paper_authors.name: Jing Xiong
paper_authors._id: 673395a92f0f0d8e0ec5af1a
paper_authors.name: Gongye Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T13:37:45.786Z
paper_authors._id: 6553316bf151de82f6a23e1d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6553316bf151de82f6a23e1d/GTBkSj4Fa3OoyM6Muz_Sc.jpeg
paper_authors.fullname: Gongye Liu
paper_authors.type: user
paper_authors.user: liuhuohuo
paper_authors._id: 673395a92f0f0d8e0ec5af1b
paper_authors.name: Lun Huang
paper_authors._id: 673395a92f0f0d8e0ec5af1c
paper_authors.name: Chengyue Wu
paper_authors._id: 673395a92f0f0d8e0ec5af1d
paper_authors.name: Taiqiang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:45:08.397Z
paper_authors._id: 6621cea88850e38ffbb1854f
paper_authors.avatarUrl: /avatars/6d73d947046faa32260ee325069976d9.svg
paper_authors.fullname: Taki WU
paper_authors.type: user
paper_authors.user: taki555
paper_authors._id: 673395a92f0f0d8e0ec5af1e
paper_authors.name: Yao Mu
paper_authors._id: 673395a92f0f0d8e0ec5af1f
paper_authors.name: Yuan Yao
paper_authors._id: 673395a92f0f0d8e0ec5af20
paper_authors.name: Hui Shen
paper_authors._id: 673395a92f0f0d8e0ec5af21
paper_authors.name: Zhongwei Wan
paper_authors._id: 673395a92f0f0d8e0ec5af22
paper_authors.name: Jinfa Huang
paper_authors._id: 673395a92f0f0d8e0ec5af23
paper_authors.name: Chaofan Tao
paper_authors._id: 673395a92f0f0d8e0ec5af24
paper_authors.name: Shen Yan
paper_authors._id: 673395a92f0f0d8e0ec5af25
paper_authors.name: Huaxiu Yao
paper_authors._id: 673395a92f0f0d8e0ec5af26
paper_authors.name: Lingpeng Kong
paper_authors._id: 673395a92f0f0d8e0ec5af27
paper_authors.name: Hongxia Yang
paper_authors._id: 673395a92f0f0d8e0ec5af28
paper_authors.name: Mi Zhang
paper_authors._id: 673395a92f0f0d8e0ec5af29
paper_authors.name: Guillermo Sapiro
paper_authors._id: 673395a92f0f0d8e0ec5af2a
paper_authors.name: Jiebo Luo
paper_authors._id: 673395a92f0f0d8e0ec5af2b
paper_authors.name: Ping Luo
paper_authors._id: 673395a92f0f0d8e0ec5af2c
paper_authors.name: Ngai Wong
paper_title: Autoregressive Models in Vision: A Survey
paper_summary: Autoregressive modeling has been a huge success in the field of natural
  language processing (NLP). Recently, autoregressive models have emerged as a
  significant area of focus in computer vision, where they excel in producing
  high-quality visual content. Autoregressive models in NLP typically operate on
  subword tokens. However, the representation strategy in computer vision can
  vary in different levels, i.e., pixel-level, token-level, or
  scale-level, reflecting the diverse and hierarchical nature of visual data
  compared to the sequential structure of language. This survey comprehensively
  examines the literature on autoregressive models applied to vision. To improve
  readability for researchers from diverse research backgrounds, we start with
  preliminary sequence representation and modeling in vision. Next, we divide the
  fundamental frameworks of visual autoregressive models into three general
  sub-categories, including pixel-based, token-based, and scale-based models
  based on the strategy of representation. We then explore the interconnections
  between autoregressive models and other generative models. Furthermore, we
  present a multi-faceted categorization of autoregressive models in computer
  vision, including image generation, video generation, 3D generation, and
  multi-modal generation. We also elaborate on their applications in diverse
  domains, including emerging domains such as embodied AI and 3D medical AI, with
  about 250 related references. Finally, we highlight the current challenges to
  autoregressive models in vision with suggestions about potential research
  directions. We have also set up a Github repository to organize the papers
  included in this survey at:
  https://github.com/ChaofanTao/Autoregressive-Models-in-Vision-Survey.
paper_ai_summary: This survey explores the application of autoregressive models in computer vision, categorizing them into pixel-based, token-based, and scale-based models, and discussing their integration with other generative models and applications in emerging domains.
paper_ai_keywords: autoregressive modeling pixel-level token-level scale-level visual autoregressive models image generation video generation 3D generation multi-modal generation embodied AI 3D medical AI