Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
Vision transformers (ViTs) have significantly changed the computer vision
landscape and have periodically exhibited superior performance in vision tasks
compared to convolutional neural networks (CNNs). Although the jury is still
out on which model type is superior, each has unique inductive biases that
shape their learning and generalization performance. For example, ViTs have
interesting properties with respect to early layer non-local feature
dependence, as well as self-attention mechanisms which enhance learning
flexibility, enabling them to ignore out-of-context image information more
effectively. We hypothesize that this power to ignore out-of-context
information (which we name patch selectivity), while integrating
in-context information in a non-local manner in early layers, allows ViTs to
more easily handle occlusion. In this study, our aim is to see whether we can
have CNNs simulate this ability of patch selectivity by effectively
hardwiring this inductive bias using Patch Mixing data augmentation, which
consists of inserting patches from another image onto a training image and
interpolating labels between the two image classes. Specifically, we use Patch
Mixing to train state-of-the-art ViTs and CNNs, assessing its impact on their
ability to ignore out-of-context patches and handle natural occlusions. We find
that ViTs do not improve nor degrade when trained using Patch Mixing, but CNNs
acquire new capabilities to ignore out-of-context information and improve on
occlusion benchmarks, leaving us to conclude that this training method is a way
of simulating in CNNs the abilities that ViTs already possess. We will release
our Patch Mixing implementation and proposed datasets for public use. Project
page: https://arielnlee.github.io/PatchMixing/
64a22f1e7d4f8315cdbcb6fe
Ariel N. Lee
extracted_confirmed
2023-08-15T01:29:34.802Z
638bcfa91987d67b340e6c1c
https://cdn-avatars.huggingface.co/v1/production/uploads/638bcfa91987d67b340e6c1c/3tHCB_J6c4-lsEZ_zJSlp.jpeg
Ariel N. Lee
user
arielnlee
64a22f1e7d4f8315cdbcb6ff
Sarah Adel Bargal
64a22f1e7d4f8315cdbcb700
Janavi Kasera
claimed_verified
2023-07-05T18:58:05.332Z
64a487b14c10e36754a24e3d
/avatars/d21435352eb5eb7757064e4328eb7c76.svg
Janavi Kasera
user
jk1204
64a22f1e7d4f8315cdbcb701
Stan Sclaroff
64a22f1e7d4f8315cdbcb702
Kate Saenko
64a22f1e7d4f8315cdbcb703
Nataniel Ruiz
extracted_confirmed
2023-07-03T15:02:28.470Z
6318c6e6cb116eab31d70a16
https://cdn-avatars.huggingface.co/v1/production/uploads/6318c6e6cb116eab31d70a16/yAw5XtvqaPUkPj2B6JOzz.jpeg
Nataniel
user
flavoredquark
Hardwiring ViT Patch Selectivity into CNNs using Patch Mixing
Vision transformers (ViTs) have significantly changed the computer vision
landscape and have periodically exhibited superior performance in vision tasks
compared to convolutional neural networks (CNNs). Although the jury is still
out on which model type is superior, each has unique inductive biases that
shape their learning and generalization performance. For example, ViTs have
interesting properties with respect to early layer non-local feature
dependence, as well as self-attention mechanisms which enhance learning
flexibility, enabling them to ignore out-of-context image information more
effectively. We hypothesize that this power to ignore out-of-context
information (which we name patch selectivity), while integrating
in-context information in a non-local manner in early layers, allows ViTs to
more easily handle occlusion. In this study, our aim is to see whether we can
have CNNs simulate this ability of patch selectivity by effectively
hardwiring this inductive bias using Patch Mixing data augmentation, which
consists of inserting patches from another image onto a training image and
interpolating labels between the two image classes. Specifically, we use Patch
Mixing to train state-of-the-art ViTs and CNNs, assessing its impact on their
ability to ignore out-of-context patches and handle natural occlusions. We find
that ViTs do not improve nor degrade when trained using Patch Mixing, but CNNs
acquire new capabilities to ignore out-of-context information and improve on
occlusion benchmarks, leaving us to conclude that this training method is a way
of simulating in CNNs the abilities that ViTs already possess. We will release
our Patch Mixing implementation and proposed datasets for public use. Project
page: https://arielnlee.github.io/PatchMixing/
Patch Mixing augmentation allows CNNs to simulate patch selectivity and handle occlusions better, similar to ViTs.
vision transformers convolutional neural networks non-local feature dependence self-attention mechanisms patch selectivity Patch Mixing inductive bias occlusion benchmarks