  Attention-based vision models, such as Vision Transformer (ViT) and its
variants, have shown promising performance in various computer vision tasks.
However, these emerging architectures suffer from large model sizes and high
computational costs, calling for efficient model compression solutions. To
date, pruning ViTs has been well studied, while other compression strategies
that have been widely applied in CNN compression, e.g., model factorization, is
little explored in the context of ViT compression. This paper explores an
efficient method for compressing vision transformers to enrich the toolset for
obtaining compact attention-based vision models. Based on the new insight on
the multi-head attention layer, we develop a highly efficient ViT compression
solution, which outperforms the state-of-the-art pruning methods. For
compressing DeiT-small and DeiT-base models on ImageNet, our proposed approach
can achieve 0.45% and 0.76% higher top-1 accuracy even with fewer parameters.
Our finding can also be applied to improve the customization efficiency of
text-to-image diffusion models, with much faster training (up to $2.6\times$
speedup) and lower extra storage cost (up to $1927.5\times$ reduction) than the
existing works.

Computer Vision and Pattern Recognition;Artificial Intelligence