title: An Image is Worth More Than 16x16 Patches: Exploring Transformers on
    Individual Pixels
summary: This work does not introduce a new method. Instead, we present an interesting
  finding that questions the necessity of the inductive bias -- locality in
  modern computer vision architectures. Concretely, we find that vanilla
  Transformers can operate by directly treating each individual pixel as a token
  and achieve highly performant results. This is substantially different from the
  popular design in Vision Transformer, which maintains the inductive bias from
  ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a
  token). We mainly showcase the effectiveness of pixels-as-tokens across three
  well-studied tasks in computer vision: supervised learning for object
  classification, self-supervised learning via masked autoencoding, and image
  generation with diffusion models. Although directly operating on individual
  pixels is less computationally practical, we believe the community must be
  aware of this surprising piece of knowledge when devising the next generation
  of neural architectures for computer vision.
paper_authors._id: 666ba4cc3814e415474a9064
paper_authors.name: Duy-Kien Nguyen
paper_authors._id: 666ba4cc3814e415474a9065
paper_authors.name: Mahmoud Assran
paper_authors._id: 666ba4cc3814e415474a9066
paper_authors.name: Unnat Jain
paper_authors._id: 666ba4cc3814e415474a9067
paper_authors.name: Martin R. Oswald
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T09:40:04.008Z
paper_authors._id: 651ed083e3c8628055890d9b
paper_authors.avatarUrl: /avatars/4cb4d3361cfd7f205c82c6e15caa11f6.svg
paper_authors.fullname: Martin Oswald
paper_authors.type: user
paper_authors.user: martin-r-oswald
paper_authors._id: 666ba4cc3814e415474a9068
paper_authors.name: Cees G. M. Snoek
paper_authors._id: 666ba4cc3814e415474a9069
paper_authors.name: Xinlei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-14T09:40:17.429Z
paper_authors._id: 63e58e3a006a775275e59e41
paper_authors.avatarUrl: /avatars/75262a35b27a2ae1939df9118120d99e.svg
paper_authors.fullname: Xinlei Chen
paper_authors.type: user
paper_authors.user: endernewton
paper_title: An Image is Worth More Than 16x16 Patches: Exploring Transformers on
    Individual Pixels
paper_summary: This work does not introduce a new method. Instead, we present an interesting
  finding that questions the necessity of the inductive bias -- locality in
  modern computer vision architectures. Concretely, we find that vanilla
  Transformers can operate by directly treating each individual pixel as a token
  and achieve highly performant results. This is substantially different from the
  popular design in Vision Transformer, which maintains the inductive bias from
  ConvNets towards local neighborhoods (e.g. by treating each 16x16 patch as a
  token). We mainly showcase the effectiveness of pixels-as-tokens across three
  well-studied tasks in computer vision: supervised learning for object
  classification, self-supervised learning via masked autoencoding, and image
  generation with diffusion models. Although directly operating on individual
  pixels is less computationally practical, we believe the community must be
  aware of this surprising piece of knowledge when devising the next generation
  of neural architectures for computer vision.
paper_ai_keywords: Transformers pixels-as-tokens inductive bias local neighborhoods 16x16 patch supervised learning object classification self-supervised learning masked Autoencoder diffusion models image generation