title: Multimodal Autoregressive Pre-training of Large Vision Encoders
summary: We introduce a novel method for pre-training of large-scale vision encoders.
  Building on recent advancements in autoregressive pre-training of vision
  models, we extend this framework to a multimodal setting, i.e., images and
  text. In this paper, we present AIMV2, a family of generalist vision encoders
  characterized by a straightforward pre-training process, scalability, and
  remarkable performance across a range of downstream tasks. This is achieved by
  pairing the vision encoder with a multimodal decoder that autoregressively
  generates raw image patches and text tokens. Our encoders excel not only in
  multimodal evaluations but also in vision benchmarks such as localization,
  grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5%
  accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently
  outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in
  multimodal image understanding across diverse settings.
paper_authors._id: 67400d6480c7c8e332a60bc3
paper_authors.name: Enrico Fini
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T10:01:48.460Z
paper_authors._id: 639f6ebb9f1f2baab2f69513
paper_authors.avatarUrl: /avatars/adb9d4feaf0a6184559b049bfb96ca0d.svg
paper_authors.fullname: Enrico Fini
paper_authors.type: user
paper_authors.user: efini
paper_authors._id: 67400d6480c7c8e332a60bc4
paper_authors.name: Mustafa Shukor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:07:33.995Z
paper_authors._id: 62bdeedd01dc22b4d22a371e
paper_authors.avatarUrl: /avatars/6adc904fb1e08661d293a966270afabb.svg
paper_authors.fullname: Mustafa Shukor
paper_authors.type: user
paper_authors.user: mshukor
paper_authors._id: 67400d6480c7c8e332a60bc5
paper_authors.name: Xiujun Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:07:42.906Z
paper_authors._id: 6254aa49d21e4cc386b6e6f9
paper_authors.avatarUrl: /avatars/ed2ea6dcdf65455cdaaeff4970eddbcd.svg
paper_authors.fullname: Xiujun Li
paper_authors.type: user
paper_authors.user: xjli
paper_authors._id: 67400d6480c7c8e332a60bc6
paper_authors.name: Philipp Dufter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:07:48.177Z
paper_authors._id: 6079980b88160e14e4e2e49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
paper_authors.fullname: Philipp Dufter
paper_authors.type: user
paper_authors.user: pdufter
paper_authors._id: 67400d6480c7c8e332a60bc7
paper_authors.name: Michal Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:07:53.273Z
paper_authors._id: 65a0ff9385f9cc255632c1c6
paper_authors.avatarUrl: /avatars/e2884cf99771247408f0a2f0edf58c92.svg
paper_authors.fullname: Michal Klein
paper_authors.type: user
paper_authors.user: michalk8
paper_authors._id: 67400d6480c7c8e332a60bc8
paper_authors.name: David Haldimann
paper_authors._id: 67400d6480c7c8e332a60bc9
paper_authors.name: Sai Aitharaju
paper_authors._id: 67400d6480c7c8e332a60bca
paper_authors.name: Victor Guilherme Turrisi da Costa
paper_authors._id: 67400d6480c7c8e332a60bcb
paper_authors.name: Louis Béthune
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:08:41.677Z
paper_authors._id: 662a8d7faa7781f8ea521f16
paper_authors.avatarUrl: /avatars/c05f9a7885b89c8818d52b98b52ecc7b.svg
paper_authors.fullname: Louis Bethune
paper_authors.type: user
paper_authors.user: AlgueRythme
paper_authors._id: 67400d6480c7c8e332a60bcc
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:08:59.126Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 67400d6480c7c8e332a60bcd
paper_authors.name: Alexander T Toshev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:09:08.756Z
paper_authors._id: 6541636c4f939214d3a0c742
paper_authors.avatarUrl: /avatars/c9507d6ec2b81681461837c8261703c7.svg
paper_authors.fullname: Alexander Toshev
paper_authors.type: user
paper_authors.user: toshev
paper_authors._id: 67400d6480c7c8e332a60bce
paper_authors.name: Marcin Eichner
paper_authors._id: 67400d6480c7c8e332a60bcf
paper_authors.name: Moin Nabi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:09:20.776Z
paper_authors._id: 5fe36e9297b72d62ab9b90cd
paper_authors.avatarUrl: /avatars/edfea7e8c86dcf355241f5a48365b35a.svg
paper_authors.fullname: Moin Nabi
paper_authors.type: user
paper_authors.user: moinnabi
paper_authors._id: 67400d6480c7c8e332a60bd0
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:09:26.423Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_authors._id: 67400d6480c7c8e332a60bd1
paper_authors.name: Joshua M. Susskind
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T13:09:32.559Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_authors._id: 67400d6480c7c8e332a60bd2
paper_authors.name: Alaaeldin El-Nouby
paper_title: Multimodal Autoregressive Pre-training of Large Vision Encoders
paper_summary: We introduce a novel method for pre-training of large-scale vision encoders.
  Building on recent advancements in autoregressive pre-training of vision
  models, we extend this framework to a multimodal setting, i.e., images and
  text. In this paper, we present AIMV2, a family of generalist vision encoders
  characterized by a straightforward pre-training process, scalability, and
  remarkable performance across a range of downstream tasks. This is achieved by
  pairing the vision encoder with a multimodal decoder that autoregressively
  generates raw image patches and text tokens. Our encoders excel not only in
  multimodal evaluations but also in vision benchmarks such as localization,
  grounding, and classification. Notably, our AIMV2-3B encoder achieves 89.5%
  accuracy on ImageNet-1k with a frozen trunk. Furthermore, AIMV2 consistently
  outperforms state-of-the-art contrastive models (e.g., CLIP, SigLIP) in
  multimodal image understanding across diverse settings.
paper_ai_keywords: autoregressive pre-training multimodal generalist vision encoders multimodal decoder raw image patches multimodal evaluations vision benchmarks localization grounding classification ImageNet-1k contrastive models CLIP SigLIP multimodal image understanding