title: HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
summary: We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR)
  visual generation model capable of directly generating 1024x1024 images,
  rivaling diffusion models in image generation quality. Existing AR models face
  limitations due to the poor image reconstruction quality of their discrete
  tokenizers and the prohibitive training costs associated with generating 1024px
  images. To address these challenges, we present the hybrid tokenizer, which
  decomposes the continuous latents from the autoencoder into two components:
  discrete tokens representing the big picture and continuous tokens representing
  the residual components that cannot be represented by the discrete tokens. The
  discrete component is modeled by a scalable-resolution discrete AR model, while
  the continuous component is learned with a lightweight residual diffusion
  module with only 37M parameters. Compared with the discrete-only VAR tokenizer,
  our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K,
  leading to a 31% generation FID improvement from 7.85 to 5.38. HART also
  outperforms state-of-the-art diffusion models in both FID and CLIP score, with
  4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced
  at https://github.com/mit-han-lab/hart.
paper_authors._id: 670e47a7ae6e15b48591a91b
paper_authors.name: Haotian Tang
paper_authors._id: 670e47a7ae6e15b48591a91c
paper_authors.name: Yecheng Wu
paper_authors._id: 670e47a7ae6e15b48591a91d
paper_authors.name: Shang Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-21T10:00:37.479Z
paper_authors._id: 641d8bacd526196afc12766d
paper_authors.avatarUrl: /avatars/73f7b2d86a7bf27940bec2b1f199d71b.svg
paper_authors.fullname: Shang Yang
paper_authors.type: user
paper_authors.user: Shangy
paper_authors._id: 670e47a7ae6e15b48591a91e
paper_authors.name: Enze Xie
paper_authors._id: 670e47a7ae6e15b48591a91f
paper_authors.name: Junsong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-21T21:33:26.122Z
paper_authors._id: 645b5b09bc7518912e1f9733
paper_authors.avatarUrl: /avatars/4d35f728b41f93881a9b67c337f4d1df.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Lawrence-cj
paper_authors._id: 670e47a7ae6e15b48591a920
paper_authors.name: Junyu Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:10:06.175Z
paper_authors._id: 646189cd5dba83471db2af58
paper_authors.avatarUrl: /avatars/d5a1549af336cb5f1fa5622250d38a73.svg
paper_authors.fullname: JUNYU CHEN
paper_authors.type: user
paper_authors.user: cjy2003
paper_authors._id: 670e47a7ae6e15b48591a921
paper_authors.name: Zhuoyang Zhang
paper_authors._id: 670e47a7ae6e15b48591a922
paper_authors.name: Han Cai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:37.556Z
paper_authors._id: 650e2b14c945dfc9386a7e28
paper_authors.avatarUrl: /avatars/0a9be20aa53f4c52a2d1a8b02d4093ea.svg
paper_authors.fullname: Han Cai
paper_authors.type: user
paper_authors.user: han-cai
paper_authors._id: 670e47a7ae6e15b48591a923
paper_authors.name: Yao Lu
paper_authors._id: 670e47a7ae6e15b48591a924
paper_authors.name: Song Han
paper_title: HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
paper_summary: We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR)
  visual generation model capable of directly generating 1024x1024 images,
  rivaling diffusion models in image generation quality. Existing AR models face
  limitations due to the poor image reconstruction quality of their discrete
  tokenizers and the prohibitive training costs associated with generating 1024px
  images. To address these challenges, we present the hybrid tokenizer, which
  decomposes the continuous latents from the autoencoder into two components:
  discrete tokens representing the big picture and continuous tokens representing
  the residual components that cannot be represented by the discrete tokens. The
  discrete component is modeled by a scalable-resolution discrete AR model, while
  the continuous component is learned with a lightweight residual diffusion
  module with only 37M parameters. Compared with the discrete-only VAR tokenizer,
  our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K,
  leading to a 31% generation FID improvement from 7.85 to 5.38. HART also
  outperforms state-of-the-art diffusion models in both FID and CLIP score, with
  4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced
  at https://github.com/mit-han-lab/hart.
paper_ai_keywords: autoregressive (AR) visual generation model image generation quality discrete tokenizers training costs technical tokenizer continuous latents autoencoder reconstruction tokens continuous tokens scalable-resolution further AR model residual diffusion module FID (Fréchet Inception Distance) CLIP score throughput MACs (multiply-accumulate operations)