title: Rejuvenating image-GPT as Strong Visual Representation Learners
summary: This paper enhances image-GPT (iGPT), one of the pioneering works that
  introduce autoregressive pretraining to predict next pixels for visual
  representation learning. Two simple yet essential changes are made. First, we
  shift the prediction target from raw pixels to semantic tokens, enabling a
  higher-level understanding of visual content. Second, we supplement the
  autoregressive modeling by instructing the model to predict not only the next
  tokens but also the visible tokens. This pipeline is particularly effective
  when semantic tokens are encoded by discriminatively trained models, such as
  CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
  showcase that D-iGPT excels as a strong learner of visual representations: A
  notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
  dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
  top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
  generalization on the downstream task and robustness on out-of-distribution
  samples. Code is avaiable at
  https://github.com/OliverRensu/D-iGPT{https://github.com/OliverRensu/D-iGPT}.
paper_authors._id: 656e9aefc542f736d82c114e
paper_authors.name: Sucheng Ren
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-28T12:18:41.576Z
paper_authors._id: 65317ea1501804124f011950
paper_authors.avatarUrl: /avatars/b055c3aba0c65d5377c69472e4576480.svg
paper_authors.fullname: Ren
paper_authors.type: user
paper_authors.user: OliverRen
paper_authors._id: 656e9aefc542f736d82c114f
paper_authors.name: Zeyu Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-14T07:26:03.407Z
paper_authors._id: 6462879777f63463fc5f75f9
paper_authors.avatarUrl: /avatars/f19d14a009e940e3b565a6c62251599b.svg
paper_authors.fullname: Zeyu Wang
paper_authors.type: user
paper_authors.user: zw123
paper_authors._id: 656e9aefc542f736d82c1150
paper_authors.name: Hongru Zhu
paper_authors._id: 656e9aefc542f736d82c1151
paper_authors.name: Junfei Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T10:11:47.610Z
paper_authors._id: 64b5ba6060274cbb296d6288
paper_authors.avatarUrl: /avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
paper_authors.fullname: Junfei Xiao
paper_authors.type: user
paper_authors.user: lambertxiao
paper_authors._id: 656e9aefc542f736d82c1152
paper_authors.name: Alan Yuille
paper_authors._id: 656e9aefc542f736d82c1153
paper_authors.name: Cihang Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-05T10:11:19.121Z
paper_authors._id: 645eb61da3c5cd8a16efffff
paper_authors.avatarUrl: /avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
paper_authors.fullname: Cihang Xie
paper_authors.type: user
paper_authors.user: cihangxie
paper_title: Rejuvenating image-GPT as Strong Visual Representation Learners
paper_summary: This paper enhances image-GPT (iGPT), one of the pioneering works that
  introduce autoregressive pretraining to predict next pixels for visual
  representation learning. Two simple yet essential changes are made. First, we
  shift the prediction target from raw pixels to semantic tokens, enabling a
  higher-level understanding of visual content. Second, we supplement the
  autoregressive modeling by instructing the model to predict not only the next
  tokens but also the visible tokens. This pipeline is particularly effective
  when semantic tokens are encoded by discriminatively trained models, such as
  CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
  showcase that D-iGPT excels as a strong learner of visual representations: A
  notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
  dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
  top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
  generalization on the downstream task and robustness on out-of-distribution
  samples. Code is avaiable at
  https://github.com/OliverRensu/D-iGPT{https://github.com/OliverRensu/D-iGPT}.
paper_ai_summary: Enhancing image-GPT through semantic token prediction and discriminatively trained models improves visual representation learning and achieves high accuracy on ImageNet-1K.
paper_ai_keywords: autoregressive pretraining semantic tokens discriminatively trained models CLIP ViT-Large top-1 accuracy