  The tremendous success of large models trained on extensive datasets
demonstrates that scale is a key ingredient in achieving superior results.
Therefore, the reflection on the rationality of designing knowledge
distillation (KD) approaches for limited-capacity architectures solely based on
small-scale datasets is now deemed imperative. In this paper, we identify the
\emph{small data pitfall} that presents in previous KD methods, which results
in the underestimation of the power of vanilla KD framework on large-scale
datasets such as ImageNet-1K. Specifically, we show that employing stronger
data augmentation techniques and using larger datasets can directly decrease
the gap between vanilla KD and other meticulously designed KD variants. This
highlights the necessity of designing and evaluating KD approaches in the
context of practical scenarios, casting off the limitations of small-scale
datasets. Our investigation of the vanilla KD and its variants in more complex
schemes, including stronger training strategies and different model capacities,
demonstrates that vanilla KD is elegantly simple but astonishingly effective in
large-scale scenarios. Without bells and whistles, we obtain state-of-the-art
ResNet-50, ViT-S, and ConvNeXtV2-T models for ImageNet, which achieve 83.1\%,
84.3\%, and 85.0\% top-1 accuracy, respectively. PyTorch code and checkpoints
can be found at https://github.com/Hao840/vanillaKD.

Computer Vision and Pattern Recognition