title: Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
    Tuning
summary: We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
  token-based, decoder-only multi-modal language model capable of generating and
  infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
  but additionally shows the extreme benefits of scaling up and tuning on more
  diverse instruction-style data. It is the first multi-modal model trained with
  a recipe adapted from text-only language models, including a large-scale
  retrieval-augmented pre-training stage and a second multi-task supervised
  fine-tuning (SFT) stage. It is also a general-purpose model that can do both
  text-to-image and image-to-text generation, allowing us to introduce
  self-contained contrastive decoding methods that produce high-quality outputs.
  Extensive experiments demonstrate that this recipe is highly effective for
  multi-modal models. CM3Leon achieves state-of-the-art performance in
  text-to-image generation with 5x less training compute than comparable methods
  (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
  unprecedented levels of controllability in tasks ranging from language-guided
  image editing to image-controlled generation and segmentation.
paper_authors._id: 64f92dc30590f3db14ba829e
paper_authors.name: Lili Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:24:07.482Z
paper_authors._id: 6466d867f3e78d1d6be05da0
paper_authors.avatarUrl: /avatars/4a2458f9fe5e1637798adfaf01cfa4fa.svg
paper_authors.fullname: Lili Yu
paper_authors.type: user
paper_authors.user: lilisierrayu
paper_authors._id: 64f92dc30590f3db14ba829f
paper_authors.name: Bowen Shi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:24:21.692Z
paper_authors._id: 6477d6e06e6c7ac608c44734
paper_authors.avatarUrl: /avatars/c6d126c5b2930531ac4b00a1ad5e6d26.svg
paper_authors.fullname: Bowen Shi
paper_authors.type: user
paper_authors.user: bowenshi
paper_authors._id: 64f92dc30590f3db14ba82a0
paper_authors.name: Ramakanth Pasunuru
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:24:35.898Z
paper_authors._id: 63b4be2334df1f9a90a1983c
paper_authors.avatarUrl: /avatars/9cdfa3492c87aabdacab684b80628f14.svg
paper_authors.fullname: Ramakanth Pasunuru
paper_authors.type: user
paper_authors.user: rpasunuru
paper_authors._id: 64f92dc30590f3db14ba82a1
paper_authors.name: Benjamin Muller
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:24:44.159Z
paper_authors._id: 5e6b7153d4cd9779932a7600
paper_authors.avatarUrl: /avatars/c1304f100758c5e2a54fd4d47f638841.svg
paper_authors.fullname: Benjamin Muller
paper_authors.type: user
paper_authors.user: benjamin-mlr
paper_authors._id: 64f92dc30590f3db14ba82a2
paper_authors.name: Olga Golovneva
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:24:53.456Z
paper_authors._id: 6318b1e06e6bd039440871d6
paper_authors.avatarUrl: /avatars/2d9f96eb4092a9d1734dcc09303b259a.svg
paper_authors.fullname: Olga Golovneva
paper_authors.type: user
paper_authors.user: Golovneva
paper_authors._id: 64f92dc30590f3db14ba82a3
paper_authors.name: Tianlu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:25:01.346Z
paper_authors._id: 63f6a0e6b4c9a104f4b95d86
paper_authors.avatarUrl: /avatars/660c3e44bea976f3e0560bf95466b03c.svg
paper_authors.fullname: Tianlu Wang
paper_authors.type: user
paper_authors.user: Tianlu
paper_authors._id: 64f92dc30590f3db14ba82a4
paper_authors.name: Arun Babu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:25:19.220Z
paper_authors._id: 61776ccc924026cc8c692630
paper_authors.avatarUrl: /avatars/44c669649402a52867861990e8b17d27.svg
paper_authors.fullname: Arun Babu
paper_authors.type: user
paper_authors.user: arbabu123
paper_authors._id: 64f92dc30590f3db14ba82a5
paper_authors.name: Binh Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:25:26.594Z
paper_authors._id: 64b8d538894eb78e39543956
paper_authors.avatarUrl: /avatars/8a85fa554097cd0b569fdc08e9169866.svg
paper_authors.fullname: Binh Tang
paper_authors.type: user
paper_authors.user: binhtang
paper_authors._id: 64f92dc30590f3db14ba82a6
paper_authors.name: Brian Karrer
paper_authors._id: 64f92dc30590f3db14ba82a7
paper_authors.name: Shelly Sheynin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:26:10.032Z
paper_authors._id: 63468e6ca6f101c7f9049132
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1665662111525-63468e6ca6f101c7f9049132.png
paper_authors.fullname: Shelly Sheynin
paper_authors.type: user
paper_authors.user: shellysheynin
paper_authors._id: 64f92dc30590f3db14ba82a8
paper_authors.name: Candace Ross
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:26:31.366Z
paper_authors._id: 628709d57cc3b7bc1b95fcf0
paper_authors.avatarUrl: /avatars/98e0d9695527125a0409d1098cdceaa4.svg
paper_authors.fullname: Candace Ross
paper_authors.type: user
paper_authors.user: candaceross
paper_authors._id: 64f92dc30590f3db14ba82a9
paper_authors.name: Adam Polyak
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:26:38.724Z
paper_authors._id: 6304d514dae2eb7d08413d62
paper_authors.avatarUrl: /avatars/02a571bc791b78d3993d9a0484b70a29.svg
paper_authors.fullname: Adam Polyak
paper_authors.type: user
paper_authors.user: adampo
paper_authors._id: 64f92dc30590f3db14ba82aa
paper_authors.name: Russell Howes
paper_authors._id: 64f92dc30590f3db14ba82ab
paper_authors.name: Vasu Sharma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T08:22:19.287Z
paper_authors._id: 65492730a3d1682f79f3ab7a
paper_authors.avatarUrl: /avatars/fc62605bdf10947dab393305c51ace96.svg
paper_authors.fullname: Vasu Sharma
paper_authors.type: user
paper_authors.user: vasusharma55
paper_authors._id: 64f92dc30590f3db14ba82ac
paper_authors.name: Puxin Xu
paper_authors._id: 64f92dc30590f3db14ba82ad
paper_authors.name: Hovhannes Tamoyan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:27:48.492Z
paper_authors._id: 63a2e8cadca41424cb87318a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a2e8cadca41424cb87318a/5YLsH5pfc__whTnVsmiQv.png
paper_authors.fullname: Hovhannes Tamoyan
paper_authors.type: user
paper_authors.user: tamohannes
paper_authors._id: 64f92dc30590f3db14ba82ae
paper_authors.name: Oron Ashual
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-13T16:15:18.478Z
paper_authors._id: 63466bd74d32f064147b9432
paper_authors.avatarUrl: /avatars/acc42481a7baa5573ecd5ec745f2b47b.svg
paper_authors.fullname: Oron Ashual
paper_authors.type: user
paper_authors.user: ashual
paper_authors._id: 64f92dc30590f3db14ba82af
paper_authors.name: Uriel Singer
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:28:42.144Z
paper_authors._id: 6345b71843f4f2d2ed113355
paper_authors.avatarUrl: /avatars/a497669a4c53a724c4f6ea615d1dda59.svg
paper_authors.fullname: Uriel Singer
paper_authors.type: user
paper_authors.user: urielsinger
paper_authors._id: 64f92dc30590f3db14ba82b0
paper_authors.name: Shang-Wen Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-28T19:31:45.042Z
paper_authors._id: 64e3a7b8e12618b261fff364
paper_authors.avatarUrl: /avatars/860a5011f150537eb923f76d354f1bc5.svg
paper_authors.fullname: Dan Li
paper_authors.type: user
paper_authors.user: swdanielli
paper_authors._id: 64f92dc30590f3db14ba82b1
paper_authors.name: Susan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:29:03.034Z
paper_authors._id: 61e88ca5ea284b407c5585f9
paper_authors.avatarUrl: /avatars/55fb192bb214e73842bd9c6f1d33d0c1.svg
paper_authors.fullname: Susan Zhang
paper_authors.type: user
paper_authors.user: suchenzang
paper_authors._id: 64f92dc30590f3db14ba82b2
paper_authors.name: Richard James
paper_authors._id: 64f92dc30590f3db14ba82b3
paper_authors.name: Gargi Ghosh
paper_authors._id: 64f92dc30590f3db14ba82b4
paper_authors.name: Yaniv Taigman
paper_authors._id: 64f92dc30590f3db14ba82b5
paper_authors.name: Maryam Fazel-Zarandi
paper_authors._id: 64f92dc30590f3db14ba82b6
paper_authors.name: Asli Celikyilmaz
paper_authors._id: 64f92dc30590f3db14ba82b7
paper_authors.name: Luke Zettlemoyer
paper_authors._id: 64f92dc30590f3db14ba82b8
paper_authors.name: Armen Aghajanyan
paper_title: Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction
    Tuning
paper_summary: We present CM3Leon (pronounced "Chameleon"), a retrieval-augmented,
  token-based, decoder-only multi-modal language model capable of generating and
  infilling both text and images. CM3Leon uses the CM3 multi-modal architecture
  but additionally shows the extreme benefits of scaling up and tuning on more
  diverse instruction-style data. It is the first multi-modal model trained with
  a recipe adapted from text-only language models, including a large-scale
  retrieval-augmented pre-training stage and a second multi-task supervised
  fine-tuning (SFT) stage. It is also a general-purpose model that can do both
  text-to-image and image-to-text generation, allowing us to introduce
  self-contained contrastive decoding methods that produce high-quality outputs.
  Extensive experiments demonstrate that this recipe is highly effective for
  multi-modal models. CM3Leon achieves state-of-the-art performance in
  text-to-image generation with 5x less training compute than comparable methods
  (zero-shot MS-COCO FID of 4.88). After SFT, CM3Leon can also demonstrate
  unprecedented levels of controllability in tasks ranging from language-guided
  image editing to image-controlled generation and segmentation.
paper_ai_keywords: retrieval-augmented decoder-only multi-modal language model CM3 multi-modal architecture instruction-style data multi-task supervised fine-tuning (SFT) text-to-image generation image-to-text generation contrastive decoding methods state-of-the-art zero-shot MS-COCO FID language-guided image editing image-controlled generation segmentation