Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and
  Resolution
The ubiquitous and demonstrably suboptimal choice of resizing images to a
fixed resolution before processing them with computer vision models has not yet
been successfully challenged. However, models such as the Vision Transformer
(ViT) offer flexible sequence-based modeling, and hence varying input sequence
lengths. We take advantage of this with NaViT (Native Resolution ViT) which
uses sequence packing during training to process inputs of arbitrary
resolutions and aspect ratios. Alongside flexible model usage, we demonstrate
improved training efficiency for large-scale supervised and contrastive
image-text pretraining. NaViT can be efficiently transferred to standard tasks
such as image and video classification, object detection, and semantic
segmentation and leads to improved results on robustness and fairness
benchmarks. At inference time, the input resolution flexibility can be used to
smoothly navigate the test-time cost-performance trade-off. We believe that
NaViT marks a departure from the standard, CNN-designed, input and modelling
pipeline used by most computer vision models, and represents a promising
direction for ViTs.
64af7af4d2373ae50c0a387b
Mostafa Dehghani
claimed_verified
2023-07-25T14:24:27.079Z
64bfb71301f1983a86c322d9
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/EjS7cIhGwQKRwr5GjjJbi.png
Mostafa Dehghani
user
dehghani
64af7af4d2373ae50c0a387c
Basil Mustafa
admin_assigned
2023-07-13T09:33:17.250Z
63cfcad6e23b90128c66685c
https://cdn-avatars.huggingface.co/v1/production/uploads/1674562206311-noauth.png
Basil Mustafa
user
BasilMustafa
64af7af4d2373ae50c0a387d
Josip Djolonga
64af7af4d2373ae50c0a387e
Jonathan Heek
64af7af4d2373ae50c0a387f
Matthias Minderer
admin_assigned
2023-07-13T09:33:50.377Z
649ac6e57c36fc2dc6e6b0f4
https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
Matthias Minderer
user
mjlm
64af7af4d2373ae50c0a3880
Mathilde Caron
admin_assigned
2023-07-13T09:34:08.442Z
63139a8be29fb2e86d5fe7a4
/avatars/8938ec33144e58e6f9b6def4d38179d8.svg
Mathilde Caron
user
MathildeC
64af7af4d2373ae50c0a3881
Andreas Steiner
64af7af4d2373ae50c0a3882
Joan Puigcerver
claimed_verified
2023-07-17T06:48:43.465Z
64afea8efd620c8a7ad4ebd7
https://cdn-avatars.huggingface.co/v1/production/uploads/64afea8efd620c8a7ad4ebd7/O3e9nLirWFVZ_jMN4lFdH.jpeg
Joan Puigcerver
user
joapuipe
64af7af4d2373ae50c0a3883
Robert Geirhos
claimed_verified
2025-01-17T17:22:42.904Z
673bbe0d7dfcdedd52619ec2
/avatars/531a44f05d0c738bbe3e028c76c2e948.svg
Robert Geirhos
user
rgeirhos
64af7af4d2373ae50c0a3884
Ibrahim Alabdulmohsin
admin_assigned
2023-07-13T09:35:03.263Z
630545da20668afe24860235
/avatars/5d82be2e7412bff1af15cc5eafa60b7d.svg
Ibrahim Alabdulmohsin
user
ibomohsin
64af7af4d2373ae50c0a3885
Avital Oliver
64af7af4d2373ae50c0a3886
Piotr Padlewski
admin_assigned
2023-07-13T09:35:29.156Z
64770ddafe51edaf1fc0322d
https://cdn-avatars.huggingface.co/v1/production/uploads/64770ddafe51edaf1fc0322d/jxN1_ALBmz7hn-IG8q75P.jpeg
Piotr Padlewski
user
prazek
64af7af4d2373ae50c0a3887
Alexey Gritsenko
admin_assigned
2023-07-13T09:35:46.456Z
62d8f9887b8dc0ba17271415
/avatars/12ec78d34fd849bad44217b212f31e98.svg
Alexey Gritsenko
user
AlexeyG
64af7af4d2373ae50c0a3888
Mario Lučić
64af7af4d2373ae50c0a3889
Neil Houlsby
admin_assigned
2023-07-13T09:36:16.464Z
64a2c7f263f69fb98d3bfdb4
/avatars/a362a236c0654b7605dcb7673e309335.svg
Neil Houlsby
user
neilhoulsby
Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and
  Resolution
The ubiquitous and demonstrably suboptimal choice of resizing images to a
fixed resolution before processing them with computer vision models has not yet
been successfully challenged. However, models such as the Vision Transformer
(ViT) offer flexible sequence-based modeling, and hence varying input sequence
lengths. We take advantage of this with NaViT (Native Resolution ViT) which
uses sequence packing during training to process inputs of arbitrary
resolutions and aspect ratios. Alongside flexible model usage, we demonstrate
improved training efficiency for large-scale supervised and contrastive
image-text pretraining. NaViT can be efficiently transferred to standard tasks
such as image and video classification, object detection, and semantic
segmentation and leads to improved results on robustness and fairness
benchmarks. At inference time, the input resolution flexibility can be used to
smoothly navigate the test-time cost-performance trade-off. We believe that
NaViT marks a departure from the standard, CNN-designed, input and modelling
pipeline used by most computer vision models, and represents a promising
direction for ViTs.
NaViT, a flexible Vision Transformer, processes images of arbitrary resolutions and aspect ratios, improving training efficiency and performance across various computer vision tasks.
Vision Transformer ViT sequence packing flexible sequence-based modeling large-scale supervised and contrastive image-text pretraining image classification video classification object detection semantic segmentation robustness fairness