title: Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
summary: Vision transformers (ViTs) have significantly changed the computer vision
  landscape and have periodically exhibited superior performance in vision tasks
  compared to convolutional neural networks (CNNs). Although the jury is still
  out on which model type is superior, each has unique inductive biases that
  shape their learning and generalization performance. For example, ViTs have
  interesting properties with respect to early layer non-local feature
  dependence, as well as self-attention mechanisms which enhance learning
  flexibility, enabling them to ignore out-of-context image information more
  effectively. We hypothesize that this power to ignore out-of-context
  information (which we name patch selectivity), while integrating
  in-context information in a non-local manner in early layers, allows ViTs to
  more easily handle occlusion. In this study, our aim is to see whether we can
  have CNNs simulate this ability of patch selectivity by effectively
  hardwiring this inductive bias using Patch Mixing data augmentation, which
  consists of inserting patches from another image onto a training image and
  interpolating labels between the two image classes. Specifically, we use Patch
  Mixing to train state-of-the-art ViTs and CNNs, assessing its impact on their
  ability to ignore out-of-context patches and handle natural occlusions. We find
  that ViTs do not improve nor degrade when trained using Patch Mixing, but CNNs
  acquire new capabilities to ignore out-of-context information and improve on
  occlusion benchmarks, leaving us to conclude that this training method is a way
  of simulating in CNNs the abilities that ViTs already possess. We will release
  our Patch Mixing implementation and proposed datasets for public use. Project
  page: https://arielnlee.github.io/PatchMixing/
paper_authors._id: 64a22f1e7d4f8315cdbcb6fe
paper_authors.name: Ariel N. Lee
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-08-15T01:29:34.802Z
paper_authors._id: 638bcfa91987d67b340e6c1c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
paper_authors.fullname: Ariel N. Lee
paper_authors.type: user
paper_authors.user: arielnlee
paper_authors._id: 64a22f1e7d4f8315cdbcb6ff
paper_authors.name: Sarah Adel Bargal
paper_authors._id: 64a22f1e7d4f8315cdbcb700
paper_authors.name: Janavi Kasera
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-05T18:58:05.332Z
paper_authors._id: 64a487b14c10e36754a24e3d
paper_authors.avatarUrl: /avatars/d21435352eb5eb7757064e4328eb7c76.svg
paper_authors.fullname: Janavi Kasera
paper_authors.type: user
paper_authors.user: jk1204
paper_authors._id: 64a22f1e7d4f8315cdbcb701
paper_authors.name: Stan Sclaroff
paper_authors._id: 64a22f1e7d4f8315cdbcb702
paper_authors.name: Kate Saenko
paper_authors._id: 64a22f1e7d4f8315cdbcb703
paper_authors.name: Nataniel Ruiz
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-07-03T15:02:28.470Z
paper_authors._id: 6318c6e6cb116eab31d70a16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
paper_authors.fullname: Nataniel
paper_authors.type: user
paper_authors.user: flavoredquark
paper_title: Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
paper_summary: Vision transformers (ViTs) have significantly changed the computer vision
  landscape and have periodically exhibited superior performance in vision tasks
  compared to convolutional neural networks (CNNs). Although the jury is still
  out on which model type is superior, each has unique inductive biases that
  shape their learning and generalization performance. For example, ViTs have
  interesting properties with respect to early layer non-local feature
  dependence, as well as self-attention mechanisms which enhance learning
  flexibility, enabling them to ignore out-of-context image information more
  effectively. We hypothesize that this power to ignore out-of-context
  information (which we name patch selectivity), while integrating
  in-context information in a non-local manner in early layers, allows ViTs to
  more easily handle occlusion. In this study, our aim is to see whether we can
  have CNNs simulate this ability of patch selectivity by effectively
  hardwiring this inductive bias using Patch Mixing data augmentation, which
  consists of inserting patches from another image onto a training image and
  interpolating labels between the two image classes. Specifically, we use Patch
  Mixing to train state-of-the-art ViTs and CNNs, assessing its impact on their
  ability to ignore out-of-context patches and handle natural occlusions. We find
  that ViTs do not improve nor degrade when trained using Patch Mixing, but CNNs
  acquire new capabilities to ignore out-of-context information and improve on
  occlusion benchmarks, leaving us to conclude that this training method is a way
  of simulating in CNNs the abilities that ViTs already possess. We will release
  our Patch Mixing implementation and proposed datasets for public use. Project
  page: https://arielnlee.github.io/PatchMixing/
paper_ai_summary: Patch Mixing augmentation allows CNNs to simulate patch selectivity and handle occlusions better, similar to ViTs.
paper_ai_keywords: vision transformers convolutional neural networks non-local feature dependence self-attention mechanisms patch selectivity Patch Mixing inductive bias occlusion benchmarks