Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
  Tuning
We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
token-based, decoder-only multi-modal language model capable of generating and
infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
but additionally shows the extreme benefits of scaling up and tuning on more
diverse instruction-style data. It is the first multi-modal model trained with
a recipe adapted from text-only language models, including a large-scale
retrieval-augmented pre-training stage and a second multi-task supervised
fine-tuning (SFT) stage. It is also a general-purpose model that can do both
text-to-image and image-to-text generation, allowing us to introduce
self-contained contrastive decoding methods that produce high-quality outputs.
Extensive experiments demonstrate that this recipe is highly effective for
multi-modal models. CM3Leon achieves state-of-the-art performance in
text-to-image generation with 5x less training compute than comparable methods
(zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
unprecedented levels of controllability in tasks ranging from language-guided
image editing to image-controlled generation and segmentation.
64f92dc30590f3db14ba829e
Lili Yu
admin_assigned
2023-09-07T13:24:07.482Z
6466d867f3e78d1d6be05da0
/avatars/4a2458f9fe5e1637798adfaf01cfa4fa.svg
Lili Yu
user
lilisierrayu
64f92dc30590f3db14ba829f
Bowen Shi
admin_assigned
2023-09-07T13:24:21.692Z
6477d6e06e6c7ac608c44734
/avatars/c6d126c5b2930531ac4b00a1ad5e6d26.svg
Bowen Shi
user
bowenshi
64f92dc30590f3db14ba82a0
Ramakanth Pasunuru
admin_assigned
2023-09-07T13:24:35.898Z
63b4be2334df1f9a90a1983c
/avatars/9cdfa3492c87aabdacab684b80628f14.svg
Ramakanth Pasunuru
user
rpasunuru
64f92dc30590f3db14ba82a1
Benjamin Muller
admin_assigned
2023-09-07T13:24:44.159Z
5e6b7153d4cd9779932a7600
/avatars/c1304f100758c5e2a54fd4d47f638841.svg
Benjamin Muller
user
benjamin-mlr
64f92dc30590f3db14ba82a2
Olga Golovneva
admin_assigned
2023-09-07T13:24:53.456Z
6318b1e06e6bd039440871d6
/avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
Olga Golovneva
user
Golovneva
64f92dc30590f3db14ba82a3
Tianlu Wang
admin_assigned
2023-09-07T13:25:01.346Z
63f6a0e6b4c9a104f4b95d86
/avatars/660c3e44bea976f3e0560bf95466b03c.svg
Tianlu Wang
user
Tianlu
64f92dc30590f3db14ba82a4
Arun Babu
admin_assigned
2023-09-07T13:25:19.220Z
61776ccc924026cc8c692630
/avatars/44c669649402a52867861990e8b17d27.svg
Arun Babu
user
arbabu123
64f92dc30590f3db14ba82a5
Binh Tang
admin_assigned
2023-09-07T13:25:26.594Z
64b8d538894eb78e39543956
/avatars/8a85fa554097cd0b569fdc08e9169866.svg
Binh Tang
user
binhtang
64f92dc30590f3db14ba82a6
Brian Karrer
64f92dc30590f3db14ba82a7
Shelly Sheynin
admin_assigned
2023-09-07T13:26:10.032Z
63468e6ca6f101c7f9049132
https://cdn-avatars.huggingface.co/v1/production/uploads/1665662111525-63468e6ca6f101c7f9049132.png
Shelly Sheynin
user
shellysheynin
64f92dc30590f3db14ba82a8
Candace Ross
admin_assigned
2023-09-07T13:26:31.366Z
628709d57cc3b7bc1b95fcf0
/avatars/98e0d9695527125a0409d1098cdceaa4.svg
Candace Ross
user
candaceross
64f92dc30590f3db14ba82a9
Adam Polyak
admin_assigned
2023-09-07T13:26:38.724Z
6304d514dae2eb7d08413d62
/avatars/02a571bc791b78d3993d9a0484b70a29.svg
Adam Polyak
user
adampo
64f92dc30590f3db14ba82aa
Russell Howes
64f92dc30590f3db14ba82ab
Vasu Sharma
claimed_verified
2025-01-10T08:22:19.287Z
65492730a3d1682f79f3ab7a
/avatars/fc62605bdf10947dab393305c51ace96.svg
Vasu Sharma
user
vasusharma55
64f92dc30590f3db14ba82ac
Puxin Xu
64f92dc30590f3db14ba82ad
Hovhannes Tamoyan
admin_assigned
2023-09-07T13:27:48.492Z
63a2e8cadca41424cb87318a
https://cdn-avatars.huggingface.co/v1/production/uploads/63a2e8cadca41424cb87318a/5YLsH5pfc__whTnVsmiQv.png
Hovhannes Tamoyan
user
tamohannes
64f92dc30590f3db14ba82ae
Oron Ashual
claimed_verified
2024-03-13T16:15:18.478Z
63466bd74d32f064147b9432
/avatars/acc42481a7baa5573ecd5ec745f2b47b.svg
Oron Ashual
user
ashual
64f92dc30590f3db14ba82af
Uriel Singer
admin_assigned
2023-09-07T13:28:42.144Z
6345b71843f4f2d2ed113355
/avatars/a497669a4c53a724c4f6ea615d1dda59.svg
Uriel Singer
user
urielsinger
64f92dc30590f3db14ba82b0
Shang-Wen Li
claimed_verified
2024-04-28T19:31:45.042Z
64e3a7b8e12618b261fff364
/avatars/860a5011f150537eb923f76d354f1bc5.svg
Shang-Wen Daniel Li
user
swdanielli
64f92dc30590f3db14ba82b1
Susan Zhang
admin_assigned
2023-09-07T13:29:03.034Z
61e88ca5ea284b407c5585f9
/avatars/55fb192bb214e73842bd9c6f1d33d0c1.svg
Susan Zhang
user
suchenzang
64f92dc30590f3db14ba82b2
Richard James
64f92dc30590f3db14ba82b3
Gargi Ghosh
64f92dc30590f3db14ba82b4
Yaniv Taigman
64f92dc30590f3db14ba82b5
Maryam Fazel-Zarandi
64f92dc30590f3db14ba82b6
Asli Celikyilmaz
64f92dc30590f3db14ba82b7
Luke Zettlemoyer
64f92dc30590f3db14ba82b8
Armen Aghajanyan
Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
  Tuning
We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
token-based, decoder-only multi-modal language model capable of generating and
infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
but additionally shows the extreme benefits of scaling up and tuning on more
diverse instruction-style data. It is the first multi-modal model trained with
a recipe adapted from text-only language models, including a large-scale
retrieval-augmented pre-training stage and a second multi-task supervised
fine-tuning (SFT) stage. It is also a general-purpose model that can do both
text-to-image and image-to-text generation, allowing us to introduce
self-contained contrastive decoding methods that produce high-quality outputs.
Extensive experiments demonstrate that this recipe is highly effective for
multi-modal models. CM3Leon achieves state-of-the-art performance in
text-to-image generation with 5x less training compute than comparable methods
(zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
unprecedented levels of controllability in tasks ranging from language-guided
image editing to image-controlled generation and segmentation.
CM3Leon, a multi-modal language model, achieves state-of-the-art text-to-image generation with efficient training and advanced control over image generation tasks.
retrieval-augmented decoder-only multi-modal language model CM3 architecture instruction-style data retrieval-augmented pre-training multi-task supervised fine-tuning text-to-image image-to-text contrastive decoding zero-shot MS-COCO FID language-guided image editing image-controlled generation segmentation