title: Swin-Free: Achieving Better Cross-Window Attention and Efficiency with
    Size-varying Window
summary: Transformer models have shown great potential in computer vision, following
  their success in language tasks. Swin Transformer is one of them that
  outperforms convolution-based architectures in terms of accuracy, while
  improving efficiency when compared to Vision Transformer (ViT) and its
  variants, which have quadratic complexity with respect to the input size. Swin
  Transformer features shifting windows that allows cross-window connection while
  limiting self-attention computation to non-overlapping local windows. However,
  shifting windows introduces memory copy operations, which account for a
  significant portion of its runtime. To mitigate this issue, we propose
  Swin-Free in which we apply size-varying windows across stages, instead of
  shifting windows, to achieve cross-connection among local windows. With this
  simple design change, Swin-Free runs faster than the Swin Transformer at
  inference with better accuracy. Furthermore, we also propose a few of Swin-Free
  variants that are faster than their Swin Transformer counterparts.
paper_authors._id: 649a7f597a65d6984919824f
paper_authors.name: Jinkyu Koo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-28T20:15:17.976Z
paper_authors._id: 649c556dfd9cea8366dfedf6
paper_authors.avatarUrl: /avatars/2e64ec054540202543eca5650b015784.svg
paper_authors.fullname: Jinkyu Koo
paper_authors.type: user
paper_authors.user: helix979
paper_authors._id: 649a7f597a65d69849198250
paper_authors.name: John Yang
paper_authors._id: 649a7f597a65d69849198251
paper_authors.name: Le An
paper_authors._id: 649a7f597a65d69849198252
paper_authors.name: Gwenaelle Cunha Sergio
paper_authors._id: 649a7f597a65d69849198253
paper_authors.name: Su Inn Park
paper_title: Swin-Free: Achieving Better Cross-Window Attention and Efficiency with
    Size-varying Window
paper_summary: Transformer models have shown great potential in computer vision, following
  their success in language tasks. Swin Transformer is one of them that
  outperforms convolution-based architectures in terms of accuracy, while
  improving efficiency when compared to Vision Transformer (ViT) and its
  variants, which have quadratic complexity with respect to the input size. Swin
  Transformer features shifting windows that allows cross-window connection while
  limiting self-attention computation to non-overlapping local windows. However,
  shifting windows introduces memory copy operations, which account for a
  significant portion of its runtime. To mitigate this issue, we propose
  Swin-Free in which we apply size-varying windows across stages, instead of
  shifting windows, to achieve cross-connection among local windows. With this
  simple design change, Swin-Free runs faster than the Swin Transformer at
  inference with better accuracy. Furthermore, we also propose a few of Swin-Free
  variants that are faster than their Swin Transformer counterparts.
paper_ai_keywords: Transformer models Swin Transformer convolution-based architectures Vision Transformer (ViT) quadratic complexity localized windows cross-window connection self-attention computation Swin-Free size-varying windows cross-connection inference Swin-Free variants weather counterparts