Iwin Transformer: Hierarchical Vision Transformer using Interleaved
  Windows
We introduce Iwin Transformer, a novel position-embedding-free hierarchical
vision transformer, which can be fine-tuned directly from low to high
resolution, through the collaboration of innovative interleaved window
attention and depthwise separable convolution. This approach uses attention to
connect distant tokens and applies convolution to link neighboring tokens,
enabling global information exchange within a single module, overcoming Swin
Transformer's limitation of requiring two consecutive blocks to approximate
global attention. Extensive experiments on visual benchmarks demonstrate that
Iwin Transformer exhibits strong competitiveness in tasks such as image
classification (87.4 top-1 accuracy on ImageNet-1K), semantic segmentation and
video action recognition. We also validate the effectiveness of the core
component in Iwin as a standalone module that can seamlessly replace the
self-attention module in class-conditional image generation. The concepts and
methods introduced by the Iwin Transformer have the potential to inspire future
research, like Iwin 3D Attention in video generation. The code and models are
available at https://github.com/cominder/Iwin-Transformer.
6883591b6d99b948a364d1bc
Simin Huo
admin_assigned
2025-07-25T15:09:55.852Z
6562baa7516893ddeb9ad518
https://cdn-avatars.huggingface.co/v1/production/uploads/6562baa7516893ddeb9ad518/xHyyR-F-x3UxTr3Zw2pIj.jpeg
sameen
user
cominder
6883591b6d99b948a364d1bd
Ning Li
Iwin Transformer: Hierarchical Vision Transformer using Interleaved
  Windows
We introduce Iwin Transformer, a novel position-embedding-free hierarchical
vision transformer, which can be fine-tuned directly from low to high
resolution, through the collaboration of innovative interleaved window
attention and depthwise separable convolution. This approach uses attention to
connect distant tokens and applies convolution to link neighboring tokens,
enabling global information exchange within a single module, overcoming Swin
Transformer's limitation of requiring two consecutive blocks to approximate
global attention. Extensive experiments on visual benchmarks demonstrate that
Iwin Transformer exhibits strong competitiveness in tasks such as image
classification (87.4 top-1 accuracy on ImageNet-1K), semantic segmentation and
video action recognition. We also validate the effectiveness of the core
component in Iwin as a standalone module that can seamlessly replace the
self-attention module in class-conditional image generation. The concepts and
methods introduced by the Iwin Transformer have the potential to inspire future
research, like Iwin 3D Attention in video generation. The code and models are
available at https://github.com/cominder/Iwin-Transformer.
Iwin Transformer, a hierarchical vision transformer without position embeddings, uses interleaved window attention and depthwise separable convolution to enable global information exchange, achieving competitive performance in image classification, semantic segmentation, and video action recognition.
position-embedding-free hierarchical vision transformer interleaved window attention depthwise separable convolution global information exchange Swin Transformer image classification semantic segmentation video action recognition class-conditional image generation Iwin 3D Attention