  It has been witnessed that masked image modeling (MIM) has shown a huge
potential in self-supervised learning in the past year. Benefiting from the
universal backbone vision transformer, MIM learns self-supervised visual
representations through masking a part of patches of the image while attempting
to recover the missing pixels. Most previous works mask patches of the image
randomly, which underutilizes the semantic information that is beneficial to
visual representation learning. On the other hand, due to the large size of the
backbone, most previous works have to spend much time on pre-training. In this
paper, we propose \textbf{Attention-driven Masking and Throwing Strategy}
(AMT), which could solve both problems above. We first leverage the
self-attention mechanism to obtain the semantic information of the image during
the training process automatically without using any supervised methods.
Masking strategy can be guided by that information to mask areas selectively,
which is helpful for representation learning. Moreover, a redundant patch
throwing strategy is proposed, which makes learning more efficient. As a
plug-and-play module for masked image modeling, AMT improves the linear probing
accuracy of MAE by $2.9\% \sim 5.9\%$ on CIFAR-10/100, STL-10, Tiny ImageNet,
and ImageNet-1K, and obtains an improved performance with respect to
fine-tuning accuracy of MAE and SimMIM. Moreover, this design also achieves
superior performance on downstream detection and segmentation tasks. Code is
available at https://github.com/guijiejie/AMT.

Computer Vision and Pattern Recognition;Machine Learning