title: Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and
    Resolution
summary: The ubiquitous and demonstrably suboptimal choice of resizing images to a
  fixed resolution before processing them with computer vision models has not yet
  been successfully challenged. However, models such as the Vision Transformer
  (ViT) offer flexible sequence-based modeling, and hence varying input sequence
  lengths. We take advantage of this with NaViT (Native Resolution ViT) which
  uses sequence packing during training to process inputs of arbitrary
  resolutions and aspect ratios. Alongside flexible model usage, we demonstrate
  improved training efficiency for large-scale supervised and contrastive
  image-text pretraining. NaViT can be efficiently transferred to standard tasks
  such as image and video classification, object detection, and semantic
  segmentation and leads to improved results on robustness and fairness
  benchmarks. At inference time, the input resolution flexibility can be used to
  smoothly navigate the test-time cost-performance trade-off. We believe that
  NaViT marks a departure from the standard, CNN-designed, input and modelling
  pipeline used by most computer vision models, and represents a promising
  direction for ViTs.
paper_authors._id: 64af7af4d2373ae50c0a387b
paper_authors.name: Mostafa Dehghani
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-25T14:24:27.079Z
paper_authors._id: 64bfb71301f1983a86c322d9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EjS7cIhGwQKRwr5GjjJbi.png
paper_authors.fullname: Mostafa Dehghani
paper_authors.type: user
paper_authors.user: dehghani
paper_authors._id: 64af7af4d2373ae50c0a387c
paper_authors.name: Basil Mustafa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:33:17.250Z
paper_authors._id: 63cfcad6e23b90128c66685c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
paper_authors.fullname: Basil Mustafa
paper_authors.type: user
paper_authors.user: BasilMustafa
paper_authors._id: 64af7af4d2373ae50c0a387d
paper_authors.name: Josip Djolonga
paper_authors._id: 64af7af4d2373ae50c0a387e
paper_authors.name: Jonathan Heek
paper_authors._id: 64af7af4d2373ae50c0a387f
paper_authors.name: Matthias Minderer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:33:50.377Z
paper_authors._id: 649ac6e57c36fc2dc6e6b0f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
paper_authors.fullname: Matthias Minderer
paper_authors.type: user
paper_authors.user: mjlm
paper_authors._id: 64af7af4d2373ae50c0a3880
paper_authors.name: Mathilde Caron
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:34:08.442Z
paper_authors._id: 63139a8be29fb2e86d5fe7a4
paper_authors.avatarUrl: /avatars/8938ec33144e58e6f9b6def4d38179d8.svg
paper_authors.fullname: Mathilde Caron
paper_authors.type: user
paper_authors.user: MathildeC
paper_authors._id: 64af7af4d2373ae50c0a3881
paper_authors.name: Andreas Steiner
paper_authors._id: 64af7af4d2373ae50c0a3882
paper_authors.name: Joan Puigcerver
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-17T06:48:43.465Z
paper_authors._id: 64afea8efd620c8a7ad4ebd7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64afea8efd620c8a7ad4ebd7/O3e9nLirWFVZ_jMN4lFdH.jpeg
paper_authors.fullname: Joan Puigcerver
paper_authors.type: user
paper_authors.user: joapuipe
paper_authors._id: 64af7af4d2373ae50c0a3883
paper_authors.name: Robert Geirhos
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-17T17:22:42.904Z
paper_authors._id: 673bbe0d7dfcdedd52619ec2
paper_authors.avatarUrl: /avatars/531a44f05d0c738bbe3e028c76c2e948.svg
paper_authors.fullname: Robert Geirhos
paper_authors.type: user
paper_authors.user: rgeirhos
paper_authors._id: 64af7af4d2373ae50c0a3884
paper_authors.name: Ibrahim Alabdulmohsin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:35:03.263Z
paper_authors._id: 630545da20668afe24860235
paper_authors.avatarUrl: /avatars/5d82be2e7412bff1af15cc5eafa60b7d.svg
paper_authors.fullname: Ibrahim Alabdulmohsin
paper_authors.type: user
paper_authors.user: ibomohsin
paper_authors._id: 64af7af4d2373ae50c0a3885
paper_authors.name: Avital Oliver
paper_authors._id: 64af7af4d2373ae50c0a3886
paper_authors.name: Piotr Padlewski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:35:29.156Z
paper_authors._id: 64770ddafe51edaf1fc0322d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64770ddafe51edaf1fc0322d/jxN1_ALBmz7hn-IG8q75P.jpeg
paper_authors.fullname: Piotr Padlewski
paper_authors.type: user
paper_authors.user: prazek
paper_authors._id: 64af7af4d2373ae50c0a3887
paper_authors.name: Alexey Gritsenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:35:46.456Z
paper_authors._id: 62d8f9887b8dc0ba17271415
paper_authors.avatarUrl: /avatars/12ec78d34fd849bad44217b212f31e98.svg
paper_authors.fullname: Alexey Gritsenko
paper_authors.type: user
paper_authors.user: AlexeyG
paper_authors._id: 64af7af4d2373ae50c0a3888
paper_authors.name: Mario Lučić
paper_authors._id: 64af7af4d2373ae50c0a3889
paper_authors.name: Neil Houlsby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:36:16.464Z
paper_authors._id: 64a2c7f263f69fb98d3bfdb4
paper_authors.avatarUrl: /avatars/a362a236c0654b7605dcb7673e309335.svg
paper_authors.fullname: Neil Houlsby
paper_authors.type: user
paper_authors.user: neilhoulsby
paper_title: Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and
    Resolution
paper_summary: The ubiquitous and demonstrably suboptimal choice of resizing images to a
  fixed resolution before processing them with computer vision models has not yet
  been successfully challenged. However, models such as the Vision Transformer
  (ViT) offer flexible sequence-based modeling, and hence varying input sequence
  lengths. We take advantage of this with NaViT (Native Resolution ViT) which
  uses sequence packing during training to process inputs of arbitrary
  resolutions and aspect ratios. Alongside flexible model usage, we demonstrate
  improved training efficiency for large-scale supervised and contrastive
  image-text pretraining. NaViT can be efficiently transferred to standard tasks
  such as image and video classification, object detection, and semantic
  segmentation and leads to improved results on robustness and fairness
  benchmarks. At inference time, the input resolution flexibility can be used to
  smoothly navigate the test-time cost-performance trade-off. We believe that
  NaViT marks a departure from the standard, CNN-designed, input and modelling
  pipeline used by most computer vision models, and represents a promising
  direction for ViTs.
paper_ai_summary: NaViT, a flexible Vision Transformer, processes images of arbitrary resolutions and aspect ratios, improving training efficiency and performance across various computer vision tasks.
paper_ai_keywords: Vision Transformer ViT sequence packing flexible sequence-based modeling large-scale supervised and contrastive image-text pretraining image classification video classification object detection semantic segmentation robustness fairness