Comprehensive Survey of Model Compression and Speed up for Vision Transformers
<|reference_start|>Comprehensive Survey of Model Compression and Speed up for Vision Transformers: Vision Transformers (ViT) have marked a paradigm shift in computer vision, outperforming state-of-the-art models across diverse tasks. However, their practical deployment is hampered by high computational and memory demands. This study addresses the challenge by evaluating four primary model compression techniques: quantization, low-rank approximation, knowledge distillation, and pruning. We methodically analyze and compare the efficacy of these techniques and their combinations in optimizing ViTs for resource-constrained environments. Our comprehensive experimental evaluation demonstrates that these methods facilitate a balanced compromise between model accuracy and computational efficiency, paving the way for wider application in edge computing devices.<|reference_end|>
@article{chen2024comprehensive,
    title={Comprehensive Survey of Model Compression and Speed up for Vision
  Transformers},
    author={Feiyang Chen, Ziqian Luo, Lisang Zhou, Xueting Pan, Ying Jiang},
    journal={Journal of Information, Technology and Policy (2024): 1-12},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.10407},
    primaryClass={cs.CV}
}