Swin-Free: Achieving Better Cross-Window Attention and Efficiency with
  Size-varying Window
Transformer models have shown great potential in computer vision, following
their success in language tasks. Swin Transformer is one of them that
outperforms convolution-based architectures in terms of accuracy, while
improving efficiency when compared to Vision Transformer (ViT) and its
variants, which have quadratic complexity with respect to the input size. Swin
Transformer features shifting windows that allows cross-window connection while
limiting self-attention computation to non-overlapping local windows. However,
shifting windows introduces memory copy operations, which account for a
significant portion of its runtime. To mitigate this issue, we propose
Swin-Free in which we apply size-varying windows across stages, instead of
shifting windows, to achieve cross-connection among local windows. With this
simple design change, Swin-Free runs faster than the Swin Transformer at
inference with better accuracy. Furthermore, we also propose a few of Swin-Free
variants that are faster than their Swin Transformer counterparts.
649a7f597a65d6984919824f
Jinkyu Koo
claimed_verified
2023-06-28T20:15:17.976Z
649c556dfd9cea8366dfedf6
/avatars/2e64ec054540202543eca5650b015784.svg
Jinkyu Koo
user
helix979
649a7f597a65d69849198250
John Yang
649a7f597a65d69849198251
Le An
649a7f597a65d69849198252
Gwenaelle Cunha Sergio
649a7f597a65d69849198253
Su Inn Park
Swin-Free: Achieving Better Cross-Window Attention and Efficiency with
  Size-varying Window
Transformer models have shown great potential in computer vision, following
their success in language tasks. Swin Transformer is one of them that
outperforms convolution-based architectures in terms of accuracy, while
improving efficiency when compared to Vision Transformer (ViT) and its
variants, which have quadratic complexity with respect to the input size. Swin
Transformer features shifting windows that allows cross-window connection while
limiting self-attention computation to non-overlapping local windows. However,
shifting windows introduces memory copy operations, which account for a
significant portion of its runtime. To mitigate this issue, we propose
Swin-Free in which we apply size-varying windows across stages, instead of
shifting windows, to achieve cross-connection among local windows. With this
simple design change, Swin-Free runs faster than the Swin Transformer at
inference with better accuracy. Furthermore, we also propose a few of Swin-Free
variants that are faster than their Swin Transformer counterparts.
Swin-Free introduces size-varying windows to reduce memory copy operations, improving inference speed and accuracy over Swin Transformer in computer vision tasks.
Swin Transformer Vision Transformer quadratic complexity self-attention shifting windows inference cross-connection local windows Swin-Free