Scalable Pre-training of Large Autoregressive Image Models
This paper introduces AIM, a collection of vision models pre-trained with an
autoregressive objective. These models are inspired by their textual
counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling
properties. Specifically, we highlight two key findings: (1) the performance of
the visual features scale with both the model capacity and the quantity of
data, (2) the value of the objective function correlates with the performance
of the model on downstream tasks. We illustrate the practical implication of
these findings by pre-training a 7 billion parameter AIM on 2 billion images,
that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at
this scale, we observe no sign of saturation in performance, suggesting that
AIM potentially represents a new frontier for training large-scale vision
models. The pre-training of AIM is similar to the pre-training of LLMs, and
does not require any image-specific strategy to stabilize the training at
scale.
65a7733cc26011a4a7ef8b3b
Alaaeldin El-Nouby
admin_assigned
2024-01-17T09:13:15.376Z
65a03550b90a1eb50fb701fd
/avatars/8160ab72b0dd2fd468fa48289ffcc36e.svg
Alaa El-Nouby
user
alaaelnouby
65a7733cc26011a4a7ef8b3c
Michal Klein
admin_assigned
2024-01-17T09:00:10.583Z
65a0ff9385f9cc255632c1c6
/avatars/e2884cf99771247408f0a2f0edf58c92.svg
Michal Klein
user
michalk8
65a7733cc26011a4a7ef8b3d
Shuangfei Zhai
claimed_verified
2024-12-13T15:46:03.703Z
675bf04240726aac79ae8e7b
/avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
Shuangfei Zhai
user
shuangfei
65a7733cc26011a4a7ef8b3e
Miguel Angel Bautista
admin_assigned
2024-01-17T09:00:34.844Z
6303c7021dd5d3c624837d42
/avatars/b98a1a5d07db426934106d40626995a5.svg
Miguel Angel Bautista
user
mbautistamartin
65a7733cc26011a4a7ef8b3f
Alexander Toshev
admin_assigned
2024-01-17T09:00:41.730Z
6541636c4f939214d3a0c742
/avatars/c9507d6ec2b81681461837c8261703c7.svg
Alexander Toshev
user
toshev
65a7733cc26011a4a7ef8b40
Vaishaal Shankar
admin_assigned
2024-01-17T09:00:48.497Z
63118add64939fabc0108b28
/avatars/0dfbd5d7f1111ec8798e3a9bcb5f7e77.svg
vaishaal shankar
user
vaishaal
65a7733cc26011a4a7ef8b41
Joshua M Susskind
admin_assigned
2024-01-17T09:00:56.399Z
6470d2247fd7ecdbd0ec3cc9
/avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
Joshua M. Susskind
user
jsusskind
65a7733cc26011a4a7ef8b42
Armand Joulin
Scalable Pre-training of Large Autoregressive Image Models
This paper introduces AIM, a collection of vision models pre-trained with an
autoregressive objective. These models are inspired by their textual
counterparts, i.e., Large Language Models (LLMs), and exhibit similar scaling
properties. Specifically, we highlight two key findings: (1) the performance of
the visual features scale with both the model capacity and the quantity of
data, (2) the value of the objective function correlates with the performance
of the model on downstream tasks. We illustrate the practical implication of
these findings by pre-training a 7 billion parameter AIM on 2 billion images,
that achieves 84.0% on ImageNet-1k with a frozen trunk. Interestingly, even at
this scale, we observe no sign of saturation in performance, suggesting that
AIM potentially represents a new frontier for training large-scale vision
models. The pre-training of AIM is similar to the pre-training of LLMs, and
does not require any image-specific strategy to stabilize the training at
scale.
Autoregressive pre-training for vision models (AIM) scales similarly to LLMs, showing improved performance with more data and parameters, and does not exhibit performance saturation.
autoregressive objective vision models Large Language Models (LLMs) ImageNet-1k model capacity downstream tasks