Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images
<|reference_start|>Pre-training of Lightweight Vision Transformers on Small Datasets with Minimally Scaled Images: Can a lightweight Vision Transformer (ViT) match or exceed the performance of Convolutional Neural Networks (CNNs) like ResNet on small datasets with small image resolutions? This report demonstrates that a pure ViT can indeed achieve superior performance through pre-training, using a masked auto-encoder technique with minimal image scaling. Our experiments on the CIFAR-10 and CIFAR-100 datasets involved ViT models with fewer than 3.65 million parameters and a multiply-accumulate (MAC) count below 0.27G, qualifying them as 'lightweight' models. Unlike previous approaches, our method attains state-of-the-art performance among similar lightweight transformer-based architectures without significantly scaling up images from CIFAR-10 and CIFAR-100. This achievement underscores the efficiency of our model, not only in handling small datasets but also in effectively processing images close to their original scale.<|reference_end|>
@article{tan2024pre-training,
    title={Pre-training of Lightweight Vision Transformers on Small Datasets with
  Minimally Scaled Images},
    author={Jen Hong Tan},
    journal={arXiv preprint arXiv:2402.03752},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.03752},
    primaryClass={cs.CV cs.LG}
}