title: FasterViT: Fast Vision Transformers with Hierarchical Attention
summary: We design a new family of hybrid CNN-ViT neural networks, named FasterViT,
  with a focus on high image throughput for computer vision (CV) applications.
  FasterViT combines the benefits of fast local representation learning in CNNs
  and global modeling properties in ViT. Our newly introduced Hierarchical
  Attention (HAT) approach decomposes global self-attention with quadratic
  complexity into a multi-level attention with reduced computational costs. We
  benefit from efficient window-based self-attention. Each window has access to
  dedicated carrier tokens that participate in local and global representation
  learning. At a high level, global self-attentions enable the efficient
  cross-window communication at lower costs. FasterViT achieves a SOTA
  Pareto-front in terms of accuracy \vs image throughput. We have extensively
  validated its effectiveness on various CV tasks including classification,
  object detection and segmentation. We also show that HAT can be used as a
  plug-and-play module for existing networks and enhance them. We further
  demonstrate significantly faster and more accurate performance than competitive
  counterparts for images with high resolution. Code is available at
  https://github.com/NVlabs/FasterViT.
paper_authors._id: 6487e14ceec01aee99ce957d
paper_authors.name: Ali Hatamizadeh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-13T17:39:15.440Z
paper_authors._id: 64414b62603214724ebd2636
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64414b62603214724ebd2636/x9JVcJRZKZE7hdEII1JRR.jpeg
paper_authors.fullname: Ali
paper_authors.type: user
paper_authors.user: ahatamiz
paper_authors._id: 6487e14ceec01aee99ce957e
paper_authors.name: Greg Heinrich
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-10T08:03:33.528Z
paper_authors._id: 64c23e06203c836866ed0bdd
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c23e06203c836866ed0bdd/rPUrP8PqL1dhWNO6L3XwI.png
paper_authors.fullname: Greg Heinrich
paper_authors.type: user
paper_authors.user: gheinrich
paper_authors._id: 6487e14ceec01aee99ce957f
paper_authors.name: Hongxu Yin
paper_authors._id: 6487e14ceec01aee99ce9580
paper_authors.name: Andrew Tao
paper_authors._id: 6487e14ceec01aee99ce9581
paper_authors.name: Jose M. Alvarez
paper_authors._id: 6487e14ceec01aee99ce9582
paper_authors.name: Jan Kautz
paper_authors._id: 6487e14ceec01aee99ce9583
paper_authors.name: Pavlo Molchanov
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-20T20:45:49.104Z
paper_authors._id: 646d0c1c534e52f8c30500a6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/646d0c1c534e52f8c30500a6/75VH8ClbRaP75BU2ONfXE.png
paper_authors.fullname: Pavlo Molchanov
paper_authors.type: user
paper_authors.user: pmolchanov
paper_title: FasterViT: Fast Vision Transformers with Hierarchical Attention
paper_summary: We design a new family of hybrid CNN-ViT neural networks, named FasterViT,
  with a focus on high image throughput for computer vision (CV) applications.
  FasterViT combines the benefits of fast local representation learning in CNNs
  and global modeling properties in ViT. Our newly introduced Hierarchical
  Attention (HAT) approach decomposes global self-attention with quadratic
  complexity into a multi-level attention with reduced computational costs. We
  benefit from efficient window-based self-attention. Each window has access to
  dedicated carrier tokens that participate in local and global representation
  learning. At a high level, global self-attentions enable the efficient
  cross-window communication at lower costs. FasterViT achieves a SOTA
  Pareto-front in terms of accuracy \vs image throughput. We have extensively
  validated its effectiveness on various CV tasks including classification,
  object detection and segmentation. We also show that HAT can be used as a
  plug-and-play module for existing networks and enhance them. We further
  demonstrate significantly faster and more accurate performance than competitive
  counterparts for images with high resolution. Code is available at
  https://github.com/NVlabs/FasterViT.
paper_ai_keywords: hybrid CNN-ViT FasterViT Hierarchical Attention (HAT) global self-attention quadratic-complexity window-based self-attention dedicated carrier tokens local representation learning global representation learning cross-window communication SOTA Pareto-front classification object detection segmentation