title: Scalable Pre-training of Large Autoregressive Image Models
summary: This paper introduces AIM, a collection of vision models pre-trained with an
  autoregressive objective. These models are inspired by their textual
  counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling
  properties. Specifically, we highlight two key findings: (1) the performance of
  the visual features scale with both the model capacity and the quantity of
  data, (2) the value of the objective function correlates with the performance
  of the model on downstream tasks. We illustrate the practical implication of
  these findings by pre-training a 7 billion parameter AIM on 2 billion images,
  that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at
  this scale, we observe no sign of saturation in performance, suggesting that
  AIM potentially represents a new frontier for training large-scale vision
  models. The pre-training of AIM is similar to the pre-training of LLMs, and
  does not require any image-specific strategy to stabilize the training at
  scale.
paper_authors._id: 65a7733cc26011a4a7ef8b3b
paper_authors.name: Alaaeldin El-Nouby
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:13:15.376Z
paper_authors._id: 65a03550b90a1eb50fb701fd
paper_authors.avatarUrl: /avatars/8160ab72b0dd2fd468fa48289ffcc36e.svg
paper_authors.fullname: Alaa El-Nouby
paper_authors.type: user
paper_authors.user: alaaelnouby
paper_authors._id: 65a7733cc26011a4a7ef8b3c
paper_authors.name: Michal Klein
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:00:10.583Z
paper_authors._id: 65a0ff9385f9cc255632c1c6
paper_authors.avatarUrl: /avatars/e2884cf99771247408f0a2f0edf58c92.svg
paper_authors.fullname: Michal Klein
paper_authors.type: user
paper_authors.user: michalk8
paper_authors._id: 65a7733cc26011a4a7ef8b3d
paper_authors.name: Shuangfei Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:46:03.703Z
paper_authors._id: 675bf04240726aac79ae8e7b
paper_authors.avatarUrl: /avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
paper_authors.fullname: Shuangfei Zhai
paper_authors.type: user
paper_authors.user: shuangfei
paper_authors._id: 65a7733cc26011a4a7ef8b3e
paper_authors.name: Miguel Angel Bautista
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:00:34.844Z
paper_authors._id: 6303c7021dd5d3c624837d42
paper_authors.avatarUrl: /avatars/b98a1a5d07db426934106d40626995a5.svg
paper_authors.fullname: Miguel Angel Bautista
paper_authors.type: user
paper_authors.user: mbautistamartin
paper_authors._id: 65a7733cc26011a4a7ef8b3f
paper_authors.name: Alexander Toshev
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:00:41.730Z
paper_authors._id: 6541636c4f939214d3a0c742
paper_authors.avatarUrl: /avatars/c9507d6ec2b81681461837c8261703c7.svg
paper_authors.fullname: Alexander Toshev
paper_authors.type: user
paper_authors.user: toshev
paper_authors._id: 65a7733cc26011a4a7ef8b40
paper_authors.name: Vaishaal Shankar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:00:48.497Z
paper_authors._id: 63118add64939fabc0108b28
paper_authors.avatarUrl: /avatars/0dfbd5d7f1111ec8798e3a9bcb5f7e77.svg
paper_authors.fullname: vaishaal shankar
paper_authors.type: user
paper_authors.user: vaishaal
paper_authors._id: 65a7733cc26011a4a7ef8b41
paper_authors.name: Joshua M Susskind
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-17T09:00:56.399Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_authors._id: 65a7733cc26011a4a7ef8b42
paper_authors.name: Armand Joulin
paper_title: Scalable Pre-training of Large Autoregressive Image Models
paper_summary: This paper introduces AIM, a collection of vision models pre-trained with an
  autoregressive objective. These models are inspired by their textual
  counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling
  properties. Specifically, we highlight two key findings: (1) the performance of
  the visual features scale with both the model capacity and the quantity of
  data, (2) the value of the objective function correlates with the performance
  of the model on downstream tasks. We illustrate the practical implication of
  these findings by pre-training a 7 billion parameter AIM on 2 billion images,
  that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at
  this scale, we observe no sign of saturation in performance, suggesting that
  AIM potentially represents a new frontier for training large-scale vision
  models. The pre-training of AIM is similar to the pre-training of LLMs, and
  does not require any image-specific strategy to stabilize the training at
  scale.
paper_ai_summary: Autoregressive pre-training for vision models (AIM) scales similarly to LLMs, showing improved performance with more data and parameters, and does not exhibit performance saturation.
paper_ai_keywords: autoregressive objective vision models Large Language Models (LLMs) ImageNet-1k model capacity downstream tasks