  Automatic data augmentation (AutoAugment) strategies are indispensable in
supervised data-efficient training protocols of vision transformers, and have
led to state-of-the-art results in supervised learning. Despite the success,
its development and application on self-supervised vision transformers have
been hindered by several barriers, including the high search cost, the lack of
supervision, and the unsuitable search space. In this work, we propose
AutoView, a self-regularized adversarial AutoAugment method, to learn views for
self-supervised vision transformers, by addressing the above barriers. First,
we reduce the search cost of AutoView to nearly zero by learning views and
network parameters simultaneously in a single forward-backward step, minimizing
and maximizing the mutual information among different augmented views,
respectively. Then, to avoid information collapse caused by the lack of label
supervision, we propose a self-regularized loss term to guarantee the
information propagation. Additionally, we present a curated augmentation policy
search space for self-supervised learning, by modifying the generally used
search space designed for supervised learning. On ImageNet, our AutoView
achieves remarkable improvement over RandAug baseline (+10.2% k-NN accuracy),
and consistently outperforms sota manually tuned view policy by a clear margin
(up to +1.3% k-NN accuracy). Extensive experiments show that AutoView
pretraining also benefits downstream tasks (+1.2% mAcc on ADE20K Semantic
Segmentation and +2.8% mAP on revisited Oxford Image Retrieval benchmark) and
improves model robustness (+2.3% Top-1 Acc on ImageNet-A and +1.0% AUPR on
ImageNet-O). Code and models will be available at
https://github.com/Trent-tangtao/AutoView.

Computer Vision and Pattern Recognition