Rejuvenating image-GPT as Strong Visual Representation Learners
This paper enhances image-GPT (iGPT), one of the pioneering works that
introduce autoregressive pretraining to predict next pixels for visual
representation learning. Two simple yet essential changes are made. First, we
shift the prediction target from raw pixels to semantic tokens, enabling a
higher-level understanding of visual content. Second, we supplement the
autoregressive modeling by instructing the model to predict not only the next
tokens but also the visible tokens. This pipeline is particularly effective
when semantic tokens are encoded by discriminatively trained models, such as
CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
showcase that D-iGPT excels as a strong learner of visual representations: A
notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
generalization on the downstream task and robustness on out-of-distribution
samples. Code is avaiable at
https://github.com/OliverRensu/D-iGPT{https://github.com/OliverRensu/D-iGPT}.
656e9aefc542f736d82c114e
Sucheng Ren
claimed_verified
2025-02-28T12:18:41.576Z
65317ea1501804124f011950
/avatars/b055c3aba0c65d5377c69472e4576480.svg
Ren
user
OliverRen
656e9aefc542f736d82c114f
Zeyu Wang
claimed_verified
2024-06-14T07:26:03.407Z
6462879777f63463fc5f75f9
/avatars/f19d14a009e940e3b565a6c62251599b.svg
Zeyu Wang
user
zw123
656e9aefc542f736d82c1150
Hongru Zhu
656e9aefc542f736d82c1151
Junfei Xiao
admin_assigned
2023-12-05T10:11:47.610Z
64b5ba6060274cbb296d6288
/avatars/67e0343954dda6e92ed3f6e7976f9f87.svg
Junfei Xiao
user
lambertxiao
656e9aefc542f736d82c1152
Alan Yuille
656e9aefc542f736d82c1153
Cihang Xie
admin_assigned
2023-12-05T10:11:19.121Z
645eb61da3c5cd8a16efffff
/avatars/9112bfeed598dfabf9e077e69e09ecc9.svg
Cihang Xie
user
cihangxie
Rejuvenating image-GPT as Strong Visual Representation Learners
This paper enhances image-GPT (iGPT), one of the pioneering works that
introduce autoregressive pretraining to predict next pixels for visual
representation learning. Two simple yet essential changes are made. First, we
shift the prediction target from raw pixels to semantic tokens, enabling a
higher-level understanding of visual content. Second, we supplement the
autoregressive modeling by instructing the model to predict not only the next
tokens but also the visible tokens. This pipeline is particularly effective
when semantic tokens are encoded by discriminatively trained models, such as
CLIP. We introduce this novel approach as D-iGPT. Extensive experiments
showcase that D-iGPT excels as a strong learner of visual representations: A
notable achievement of D-iGPT is its compelling performance on the ImageNet-1K
dataset -- by training on publicly available datasets, D-iGPT achieves 89.5\%
top-1 accuracy with a vanilla ViT-Large model. This model also shows strong
generalization on the downstream task and robustness on out-of-distribution
samples. Code is avaiable at
https://github.com/OliverRensu/D-iGPT{https://github.com/OliverRensu/D-iGPT}.
Enhancing image-GPT through semantic token prediction and discriminatively trained models improves visual representation learning and achieves high accuracy on ImageNet-1K.
autoregressive pretraining semantic tokens discriminatively trained models CLIP ViT-Large top-1 accuracy